1 to 25 of 44 Apache Jobs in the South East

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. Understanding ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT. Experience working with and designing infrastructures for large‐scale ...

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure preferred ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement ...

technology partners. Knowledge of enterprise data platforms, data engineering, data governance, and analytics ecosystems. Experience supporting large-scale transformation programmes. Understanding of Power BI, Apache Superset, Grafana, and enterprise reporting solutions. Strong knowledge of CI/CD pipelines and automation technologies. Experience with Terraform, Ansible, PowerShell, Python, or similar ...

similar data-focused role Strong SQL and Python development skills Experience building and supporting scalable data pipelines and transformation processes Hands-on experience with Apache Spark, Databricks, dbt and cloud-based data platforms Good understanding of data modelling principles, data lakes and data warehouse architectures Experience working with ...

Skills MUST HAVE: Strong hands-on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong cloud ...

decisions, documentation and continuous improvement initiatives. Key Skills & Experience Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. ...

hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with emerging technologies in data engineering ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

critical data ecosystem while supporting the delivery of high-value data products. Key Responsibilities Design, build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimise batch and streaming data workflows, ensuring reliability, scalability and performance. Contribute to the design and implementation of modern Lakehouse data ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will be joining … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure data ...

cloud development. Exposure in hosting and setting up container-based applications. Experience is Api Gateway Management services Exposure to Caching Framework like (Redis, Apache Ignite etc) Exposure to Log Monitoring Tools like (Elastic Search & Splunk etc ) The ability to understand and support, modify, and maintain systems and code developed ...

Monitor and maintain - Monitor data systems for performance issues and make any necessary updates. What You’ll Bring ETL processing languages such as Python Apache Spark/NiFi/Kafka Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) AWS and data-related services Palantir Foundry ...

Experience of working with MOD Accreditors; and Knowledge of Onsite/project tech stack includes but is not limited to Azure, AWS, Docker, Kubernetes, Apache (NiFi, Kafka), NodeJS, Typescript, MongoDB, AI, Machine Learning etc. Just as important are curiosity, strong problem solving ability, and the motivation to continuously improve ...

models. Your Skills & Experience Strong experience with SQL and modern data engineering practices. Hands-on experience with Snowflake and dbt. Strong experience working with Apache Airflow for data orchestration. Experience working with AWS data services and cloud-based data platforms. Strong Python skills for data engineering and pipeline development. ...

Snowflake Hands-on expertise with Spark/PySpark and big data processing technologies Knowledge of open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints Familiarity with AI evaluation and observability practices for LLM workloads Understanding ...

Snowflake Hands-on expertise with Spark/PySpark and big data processing technologies Knowledge of open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints Familiarity with AI evaluation and observability practices for LLM workloads Understanding ...

within financial markets/securities . Experience working with GCP BigQuery . Experience with GCP Cloud SQL . Experience with Terraform . Experience with Apache Airflow . Understanding of security reference data, pricing data or corporate actions. The Project The wider project is focused on creating a multi-asset ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition and design ...

Infrastructure-as-Code using Terraform Work with containers, orchestration and troubleshooting Support Linux infrastructure and PostgreSQL Manage load balancers including AWS ALB, NGINX and Apache Troubleshoot infrastructure issues, including memory faults and server crashes Manage Active Directory, SSL certificates and secure communications Support networking across AWS, including VPCs, IGWs ...