1 to 25 of 263 Permanent Apache Jobs in England

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale, encompassing ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence ...

Python, Java or Scala, and SQL. Hands-on experience building and operating large-scale data pipelines using big data and distributed processing technologies (e.g. Apache Spark, Hadoop). Experience designing and maintaining data lakes, distributed storage platforms, and data pipeline orchestration tools. Experience working with cloud-native data platforms ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

data quality and consistency across platforms while complying with governance policies.What You Bring:Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs.Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB, Cosmos ...

automation capabilities, to improve the value realized by automation Required qualifications, skills and capabilities Experience with building data pipelines on Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT ...

technologies as our stack evolves.Nice to HaveExperience with stream processing frameworks such as Flink or Spark Streaming.Experience with task orchestration tools such as Apache Airflow or similar systems.Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT.Experience working with and designing infrastructures for large-scale data processing ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Experience working with and designing infrastructures for large‐scale data processing (Hive, Snowflake, NoSQL databases). Experience with data governance practices ...

efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries using … Apache Spark for large-scale datasets Monitor, troubleshoot, and improve data systems to minimise downtime and maximise efficiency Collaboration & Mentorship Partner with analytics, software engineering, and other teams to deliver integrated, high-quality solutions Provide technical guidance and mentorship to junior engineers, promoting best practices in software and data ...

expected to be up to 10% but may increase over time as the business evolves.Desired QualificationsExperience working with distributed data processing frameworks, including Apache Spark.Exposure to advanced analytics or AI‐adjacent data use cases, including preparing data for machine learning or intelligent applications.Familiarity with additional Azure services such ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands-on expertise ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc.Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools (e.g. ...

Advanced experience operating Delta tables at scale, covering transaction semantics, schema evolution, partitioning, and compaction, alongside highly optimised analytical query execution using DuckDB or Apache Spark.• Production-grade relational database expertise, particularly PostgreSQL (query tuning, indexing, partitioning, replication, and schema design across OLTP and analytics).• Strong KDB experience ...

measurable value.• Excellent analytical, problem-solving, and communication skills.• Proven experience leading data modernisation initiatives across multiple sectors.Technical Expertise• Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog.• Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation).• Hands-on expertise ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

testing, and deployment.Deep understanding of Quantexa components such as:Entity Resolution (ER)Scoring FrameworkContextual Network GenerationUI ConfigurationStrong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearchExperience with data modelling and design in the context of large-scale analytics platforms.Strong stakeholder management and communication skills – capable of interfacing ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark.Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms.Implement continuous integration and continuous deployment ...

design approaches.You understand where traditional patterns remain appropriate – and where modern approaches add measurable value.Data Engineering & Platform Understanding:Strong knowledge of distributed processing (e.g. Apache Spark).Familiarity with Hadoop ecosystem components.Working knowledge of SQL and NoSQL technologies.Understanding of ETL/ELT design trade-offs.Awareness of DevOps practices ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

/CD, DevOps, and Infrastructure as Code practices for data platforms. Metadata management, lineage tooling, and data observability/monitoring experience. Familiarity with dbt, Apache Airflow, Azure Data Factory, Kafka, Event Hubs, or comparable orchestration/integration tools. Familiarity with enterprise AI platforms — Azure AI, AWS SageMaker, Google Vertex ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting tools ...

design data architectures that facilitate model training and inference.Data Governance Tools: Familiarity with data governance or data cataloguing tools (such as Collibra, Alation, or Apache Atlas) and lineage-tracking systems. Hands-on experience setting up or maintaining a data catalogue, documenting data definitions, or automating data lineage capture ...

volume, data-intensive environments. Extensive AWS experience including services such as Glue, Lambda, S3, Redshift, and wider cloud-native data tooling. Strong knowledge of Apache Spark and large-scale data processing. Experience working with streaming technologies such as Kafka. Expert SQL and data modelling capabilities. Experience designing modern data ...

Data pipeline development and optimisation Lakehouse architectures ETL/ELT frameworks Enterprise data platform implementation Cloud Technologies (AWS) Experience with: Amazon S3 AWS Glue Apache Airflow IAM AWS Lambda (desirable) Preferred Experience Financial Services or Banking sector experience. Experience working within regulated environments. Large-scale cloud transformation programmes. Agile ...