20 of 20 Apache Spark Jobs in the North of England

using Docker and Kubernetes (EKS/AKS). Develop automation scripts and tooling using Python, Bash, and PowerShell. Support data and orchestration platforms using Apache Spark and Apache Airflow. Use Jupyter Notebooks for development, troubleshooting, and operational analysis. Implement monitoring, logging, and observability solutions using Prometheus, Grafana … Ansible/Chef Containers & Orchestration Docker Kubernetes (EKS/AKS) CI/CD & Automation GitLab CI/CD Python Bash PowerShell Data & Platform Engineering Apache Spark Apache Airflow Jupyter Notebooks Observability & Monitoring Prometheus Grafana Splunk Security & DevSecOps Trivy Trend Micro Wiz Security scanning and compliance automation Networking ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Swagger, alongside accessibility testing (WCAG 2.2 with assistive tech such as JAWS/Dragon), plus Infrastructure as Code and data platforms using Terraform, Apache Spark, Python/Airflow, Docker, security scanning (e.g., Trixy, Trend Micro, Wiz) and Juryter Notebooks, integrated with DWP Integration Team products (e.g.Data Access Layer … Azure cloud with Terraform for Infrastructure as Code, Java and Python services built as RESTful microservices (Spring Boot) supported by TDD/BDD, Apache Spark for data processing and Apache Airflow for orchestration, containerization with Docker, CI/CD and source control with GitLab/GitHub, streaming ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

requirements 3+ years of Python in production environments Strong AWS experience across core services Strong SQL skills for querying, transformation, and optimisation Experience with Apache Spark or serverless architectures Core responsibilities and skills Build and maintain real-time and batch ETL pipelines using Python on large, complex datasets … across large data volumes Model and transform data into well-structured, reusable datasets for downstream use Work with distributed data processing tools such as Apache Spark to handle high-volume workloads Improve existing pipelines by identifying bottlenecks and refactoring code and queries Develop and maintain data workflows, scheduling ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

Databricks Platform & Data Engineering Maintain and optimise an existing Databricks environment, ensuring reliability, performance, and cost‐effectiveness Design, build, and enhance data pipelines using Spark and Databricks best practices Develop scalable, reusable data engineering patterns to support multiple data domains Monitor and troubleshoot pipeline failures, performance bottlenecks, and data … Improvement ·Continuously improve pipeline performance, reliability, and maintainability ·Identify opportunities to automate, standardize or simplify data engineering workflows ·Stay up to date with Databricks, Spark and cloud data engineering best practices Essential Skills & Experience ·Proven experience as a Data Engineer working in a modern cloud data platform ·Strong hands ...

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

experience working with data warehouses. Solid understanding of ETL/ELT design and data pipeline optimisation. Experience with big data frameworks such as Apache Spark, and exposure to streaming technologies like Kafka. Strong problem-solving skills and the ability to collaborate effectively across teams. What's on offer ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

driven decision making. What youll be doing Building and optimising scalable data pipelines using Java (primary) and technologies like Kafka, Flink, or Spark Developing real-time streaming solutions and event-driven architectures using AWS services (Kinesis, MSK, Lambda, Glue) Contributing to data architecture, modelling, and best practices across modern ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

translate business needs into technical roadmaps What were looking for Strong experience in data architecture & large-scale systems design Expertise across Python, SQL, Spark, Scala Deep knowledge of AI/ML frameworks and modern data platforms Experience with cloud-native architecture and DevOps (Kubernetes, Terraform, CI/CD) Proven ...

manage multiple priorities and respond calmly to issues. • Self-motivated with a collaborative approach. Preferred: • Familiarity with AWS data tooling such as S3, Athena, Spark or notebooks. • Experience supporting data products or client-facing datasets. • Understanding of release testing, UAT or production support processes. • Exposure to Jira/Agile ...

expand their Devops team (multiple roles). Key requirements: Strong experience with AWS services and infrastructure as code using Terraform Data engineering expertise using Apache Spark and Python, including development in Jupyter Notebooks Workflow orchestration and containerisation experience with Apache Airflow and Docker Version control ...

optimise scalable data pipelines (batch and real-time) Work primarily with Java , with exposure to Python and streaming frameworks such as Kafka, Flink, or Spark Develop event-driven architectures and streaming integrations Integrate data from multiple sources using AWS services (e.g. Kinesis, Lambda, Glue, MSK) Architecture & Data Management Contribute … experience in Data Engineering or large-scale data systems Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark (streaming focus is a strong advantage) Solid understanding of streaming concepts (event time, state, backpressure) Experience building ETL/ELT or real-time data ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer ...

systems, and modern cloud architectures . What youll be doing Design and build scalable data pipelines (batch & streaming) Work with technologies like Kafka, Flink, Spark, AWS services Develop real-time, event-driven data solutions Contribute to data architecture and modelling best practices Implement CI/CD pipelines and Infrastructure … deliver clean datasets What were looking for Strong programming skills in Java (preferred) or Python Experience with streaming technologies (Kafka/Flink/Spark) Knowledge of distributed systems and large-scale data processing Experience with cloud platforms (AWS preferred) Familiarity with DevOps tools and containerisation (Docker/Kubernetes ...