26 to 50 of 257 Apache Jobs in London

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure preferred ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement ...

data pipelines, orchestration, and automation; Solid software engineering fundamentals, including version control, testing, and CI/CD; Experience with workflow orchestration frameworks such as Apache Airflow; Experience with cloud-native data platforms, preferably AWS or Azure; Experience with enterprise data warehouses, preferably Snowflake; Experience with performance tuning, cost controls ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create data ...

similar language. Proficiency in database technologies (SQL, NoSQL, time-series databases) and data modelling. Strong understanding of data pipeline orchestration tools (e.g., Apache Airflow, Kubernetes). You thrive when working as part of a team Comfortable in a fast-paced environment Have excellent written and verbal English skills Last ...

platformsAdvanced Python and SQL skills, with experience in modern engineering practices such as CI/CD and TDDExperience with big data frameworks such as Apache Spark or similar technologiesSolid understanding of data modelling, data warehousing, OLTP and data lake architecturesExperience working with cloud ecosystems such as AWS, Azure ...

Hands‐on experience as a Data Engineer or Data Platform Developer, including direct contributions to production codebases. Expert‐level proficiency in Python, SQL, and Apache Spark for data pipeline development. Proven experience building ELT workflows using Fivetran, Airflow, DBT, or Databricks. Solid understanding of data modeling, dimensional design ...

pipelines.QualificationsHands-on experience as a Data Engineer or Data Platform Developer, including direct contributions to production codebases.Expert-level proficiency in Python, SQL, and Apache Spark for data pipeline development.Proven experience building ELT workflows using Fivetran, Airflow, DBT, or Databricks.Solid understanding of data modeling, dimensional design, and schema normalization ...

Terraform and Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery.Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines.Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of scheduling/orchestration tools such as Autosys and/or Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift or equivalent ...

Oracle databases. Experience designing and supporting ETL/ELT pipelines and data quality frameworks (rules, thresholds, exception handling). Working knowledge of Autosys and Apache Airflow (monitoring schedules, reruns, failure triage). Experience with CI/CD and release controls (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift ...

skills. Ability to think creatively and propose innovative solutions. To qualify for this role, you must have: Proven experience and knowledge with PySpark and Apache Spark including the fundamentals of how it works. Experience with cloud platforms (e.g., AWS, Azure, GCP) and data lake architectures. Strong experience and programming ...

expertise in data pipeline development, ETL/ELT processes, and data warehousing solutions.Proficient with cloud platforms (AWS, GCP, Azure) and big data tools like Apache Spark, Python, etc.Experience with relational and NoSQL databases.Solid understanding of data modeling, data architecture, engineering frameworksProven ability to lead and scale engineering teams ...

Redshift.Experience with data transformation tools such as dbt, or Dataflow.Understanding of data modeling techniques.Experience with using visualisation platforms such as Looker, Tableau, or Apache Superset.Understanding of software engineering best practices and their application to data processing systems.Knowledge of Python, Java or Flink is a plus, but not a necessity.Strong ...

with data transformation tools such as dbt, or Dataflow. Understanding of data modeling techniques. Experience with using visualisation platforms such as Looker, Tableau, or Apache Superset. Understanding of software engineering best practices and their application to data processing systems. Knowledge of Python, Java or Flink is a plus ...

using Databricks REST APIs, Prometheus, or Azure Monitor.• Maintain best practices for data engineering performance, testing, and documentation.Technical Skills Required• Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake.• Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies.• Experience tuning job configurations (e.g. ...

Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python ...

experience in Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats.Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python ...

knowledge of optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow … cost optimisation strategies for cloud data platforms Experience with data quality frameworks and implementation Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Experience with ...

Skills MUST HAVE: Strong hands‐on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong cloud ...

skills; experience with Oracle databases . Experience designing and supporting ETL/ELT pipelines and data quality frameworks . Working knowledge of Autosys& Apache Airflow Experience with CI/CD tools (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift) Familiarity with AWS S3 for large-scale data storage. Experience ...

datasets— hands-on familiarity with vessel-tracking data, its structure, quirks, and limitations.• SQL— strong proficiency, including query optimization on large datasets.Nice-to-Haves• Apache Airflow (or equivalent orchestration) for scheduling and managing pipelines.• AWS experience (Azure or GCP equally welcome).• Machine Learning experience, particularly applied to geospatial ...

decisions, documentation and continuous improvement initiatives. Key Skills & Experience Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. ...

scale data pipelines, with a strong grasp of data structuring for performance and efficiencyHands-on experience with Airflow for orchestrating complex data workflows, and Apache Iceberg for table format management at scaleExtensive AWS experience including EKS, Docker, and Kubernetes; comfortable owning infrastructure deployment and scaling decisionsCI/CD fluency ...