1 to 25 of 108 Apache Spark Jobs in the UK

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

using Docker and Kubernetes (EKS/AKS). Develop automation scripts and tooling using Python, Bash, and PowerShell. Support data and orchestration platforms using Apache Spark and Apache Airflow. Use Jupyter Notebooks for development, troubleshooting, and operational analysis. Implement monitoring, logging, and observability solutions using Prometheus, Grafana … Ansible/Chef Containers & Orchestration Docker Kubernetes (EKS/AKS) CI/CD & Automation GitLab CI/CD Python Bash PowerShell Data & Platform Engineering Apache Spark Apache Airflow Jupyter Notebooks Observability & Monitoring Prometheus Grafana Splunk Security & DevSecOps Trivy Trend Micro Wiz Security scanning and compliance automation Networking ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via chelsea.hackett@searchability.com ABOUT THE CLIENT Due to continued growth, we’re seeking a skilled Data … optimise scalable data pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Swagger, alongside accessibility testing (WCAG 2.2 with assistive tech such as JAWS/Dragon), plus Infrastructure as Code and data platforms using Terraform, Apache Spark, Python/Airflow, Docker, security scanning (e.g., Trixy, Trend Micro, Wiz) and Juryter Notebooks, integrated with DWP Integration Team products (e.g.Data Access Layer … Azure cloud with Terraform for Infrastructure as Code, Java and Python services built as RESTful microservices (Spring Boot) supported by TDD/BDD, Apache Spark for data processing and Apache Airflow for orchestration, containerization with Docker, CI/CD and source control with GitLab/GitHub, streaming ...

code. Perfect if you want scope for growth without going “post-technical.” What you’ll do Design and build modern data platforms using Databricks, Apache Spark, Snowflake, and cloud-native services (AWS, Azure, or GCP). Develop robust pipelines for real-time and batch data ingestion from diverse … Lead Data Engineer in complex enterprise environments. Can demonstrate strong coding skills in Python (Scala or functional languages a plus). Expertise with Databricks, Apache Spark, and Snowflake is a must (HDFS/HBase also useful). Has experience integrating large, messy datasets into reliable, scalable data products. ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

partnership with Inspire People, is hiring a Lead Data Engineer with strong SQL and Python expertise, experience across modern cloud data platforms including Databricks, Apache Spark and lakehouse architectures, and a proven ability to design, build and lead delivery of reliable data products. This is a permanent role … Strong proficiency in SQL and Python - Experience with modern cloud data platforms and lakehouse architectures - Hands-on experience with big data technologies such as Apache Spark or Databricks - Strong knowledge of data warehousing concepts - Experience with build, test and deployment automation - Proven leadership capability leading or mentoring data ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

using assistive technologies such as JAWS and Dragon. Support Infrastructure as Code (IaC) initiatives using Terraform and Docker. Work with data engineering tools including Apache Spark, Python, Airflow, and Jupyter Notebooks. Participate in security and vulnerability scanning using tools like Trend Micro, Wiz, and Trivy/Trixy. Collaborate … frameworks Cloud & DevOps AWS Azure exposure GitLab Jenkins/Concourse Docker Terraform Testing & Quality TDD/BDD Playwright Selenium Cucumber Mockito Swagger Data & Integration Apache Spark Python Airflow Jupyter Notebooks Security & Accessibility WCAG 2.2 compliance Accessibility testing tools (JAWS, Dragon) Security scanning tools such as Trend Micro ...

requirements 3+ years of Python in production environments Strong AWS experience across core services Strong SQL skills for querying, transformation, and optimisation Experience with Apache Spark or serverless architectures Core responsibilities and skills Build and maintain real-time and batch ETL pipelines using Python on large, complex datasets … across large data volumes Model and transform data into well-structured, reusable datasets for downstream use Work with distributed data processing tools such as Apache Spark to handle high-volume workloads Improve existing pipelines by identifying bottlenecks and refactoring code and queries Develop and maintain data workflows, scheduling ...

ESSENTIAL CRITERIA: Strong experience designing and operating large-scale distributed or event-driven systems. Hands-on experience with streaming and data technologies such as Apache Kafka, Apache Flink, Spark, or similar platforms. Experience building scalable cloud-based services, ideally within AWS environments and modern backend technologies such … used across our teams include: Java/Spring Boot and TypeScript/JavaScript AWS services including MSK, Lambda, ECS, S3, DynamoDB, Redshift, and SQS Apache Kafka, Apache Flink, Airflow, Spark, and EventBridge SQL and NoSQL databases including Postgres, MongoDB, DynamoDB, and Timestream CI/CD and automation ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

delivery of data products. Develop scalable cloud-native solutions using AWS technologies. Build and maintain backend services and infrastructure using Python. Work with Apache Spark and Apache Airflow for data orchestration pipelines Implement infrastructure as code using Terraform and/or AWS CloudFormation . Manage IAM setups … experience focused on AWS-based data platforms. Strong experience with DevOps, cloud engineering and running production data platforms. Hands-on experience provisioning and managing Apache Airflow . Experience with CI/CD pipelines, automated testing and agile software development practices. Comfortable working with large and complex datasets while breaking ...

streaming data Develop data solutions using Databricks Lakehouse and Delta Lake Ingest and integrate data from internal and external sources (e.g. Kafka, CDC) Optimise Spark jobs and data workflows for performance, scalability, and cost efficiency Manage infrastructure and environments using Terraform (IaC) Ensure data quality, monitoring, and reliability Implement … Demonstrated experience in data engineering, with a proven ability to build scalable data solutions Strong proficiency in Python and SQL Hands-on experience with Apache Spark (including Structured Streaming) Experience with Databricks (Workflows, Delta Live Tables, Lakehouse architecture) Experience with cloud platforms (AWS, Azure, or GCP) Experience with ...

while also ensuring the stability and reliability of existing workloads. Key Responsibilities Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL Develop and manage Delta Lake-based data models, supporting a structured Lakehouse/Medallion architecture (Bronze, Silver, Gold layers) Support the ingestion … downstream consumption Required Skills & Experience Strong commercial experience working with Azure Databricks as a core data processing platform Deep expertise in: PySpark/Apache Spark for distributed data processing Delta Lake (ACID transactions, optimisation, data versioning) Spark SQL and advanced SQL techniques Python for data engineering ...

while also ensuring the stability and reliability of existing workloads. Key Responsibilities Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL Develop and manage Delta Lake-based data models , supporting a structured Lakehouse/Medallion architecture (Bronze, Silver, Gold layers) Support the ingestion … downstream consumption Required Skills & Experience Strong commercial experience working with Azure Databricks as a core data processing platform Deep expertise in: PySpark/Apache Spark for distributed data processing Delta Lake (ACID transactions, optimisation, data versioning) Spark SQL and advanced SQL techniques Python for data engineering ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

Databricks Platform & Data Engineering Maintain and optimise an existing Databricks environment, ensuring reliability, performance, and cost‐effectiveness Design, build, and enhance data pipelines using Spark and Databricks best practices Develop scalable, reusable data engineering patterns to support multiple data domains Monitor and troubleshoot pipeline failures, performance bottlenecks, and data … Improvement ·Continuously improve pipeline performance, reliability, and maintainability ·Identify opportunities to automate, standardize or simplify data engineering workflows ·Stay up to date with Databricks, Spark and cloud data engineering best practices Essential Skills & Experience ·Proven experience as a Data Engineer working in a modern cloud data platform ·Strong hands ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...