1 to 25 of 48 Apache Spark Jobs in the UK excluding London

using Docker and Kubernetes (EKS/AKS). Develop automation scripts and tooling using Python, Bash, and PowerShell. Support data and orchestration platforms using Apache Spark and Apache Airflow. Use Jupyter Notebooks for development, troubleshooting, and operational analysis. Implement monitoring, logging, and observability solutions using Prometheus, Grafana … Ansible/Chef Containers & Orchestration Docker Kubernetes (EKS/AKS) CI/CD & Automation GitLab CI/CD Python Bash PowerShell Data & Platform Engineering Apache Spark Apache Airflow Jupyter Notebooks Observability & Monitoring Prometheus Grafana Splunk Security & DevSecOps Trivy Trend Micro Wiz Security scanning and compliance automation Networking ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Swagger, alongside accessibility testing (WCAG 2.2 with assistive tech such as JAWS/Dragon), plus Infrastructure as Code and data platforms using Terraform, Apache Spark, Python/Airflow, Docker, security scanning (e.g., Trixy, Trend Micro, Wiz) and Juryter Notebooks, integrated with DWP Integration Team products (e.g.Data Access Layer … Azure cloud with Terraform for Infrastructure as Code, Java and Python services built as RESTful microservices (Spring Boot) supported by TDD/BDD, Apache Spark for data processing and Apache Airflow for orchestration, containerization with Docker, CI/CD and source control with GitLab/GitHub, streaming ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

partnership with Inspire People, is hiring a Lead Data Engineer with strong SQL and Python expertise, experience across modern cloud data platforms including Databricks, Apache Spark and lakehouse architectures, and a proven ability to design, build and lead delivery of reliable data products. This is a permanent role … Strong proficiency in SQL and Python - Experience with modern cloud data platforms and lakehouse architectures - Hands-on experience with big data technologies such as Apache Spark or Databricks - Strong knowledge of data warehousing concepts - Experience with build, test and deployment automation - Proven leadership capability leading or mentoring data ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

requirements 3+ years of Python in production environments Strong AWS experience across core services Strong SQL skills for querying, transformation, and optimisation Experience with Apache Spark or serverless architectures Core responsibilities and skills Build and maintain real-time and batch ETL pipelines using Python on large, complex datasets … across large data volumes Model and transform data into well-structured, reusable datasets for downstream use Work with distributed data processing tools such as Apache Spark to handle high-volume workloads Improve existing pipelines by identifying bottlenecks and refactoring code and queries Develop and maintain data workflows, scheduling ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

while also ensuring the stability and reliability of existing workloads. Key Responsibilities Design, build, and optimise data pipelines within Azure Databricks using PySpark and Spark SQL Develop and manage Delta Lake-based data models , supporting a structured Lakehouse/Medallion architecture (Bronze, Silver, Gold layers) Support the ingestion … downstream consumption Required Skills & Experience Strong commercial experience working with Azure Databricks as a core data processing platform Deep expertise in: PySpark/Apache Spark for distributed data processing Delta Lake (ACID transactions, optimisation, data versioning) Spark SQL and advanced SQL techniques Python for data engineering ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

experience in data engineering or large-scale data platforms Strong programming skills in Java (preferred) or Python Hands-on experience with Kafka, Flink, or Spark, with a preference for streaming technologies Solid understanding of real-time data processing concepts such as event time, state handling, and backpressure Experience building ...

Databricks Platform & Data Engineering Maintain and optimise an existing Databricks environment, ensuring reliability, performance, and cost‐effectiveness Design, build, and enhance data pipelines using Spark and Databricks best practices Develop scalable, reusable data engineering patterns to support multiple data domains Monitor and troubleshoot pipeline failures, performance bottlenecks, and data … Improvement ·Continuously improve pipeline performance, reliability, and maintainability ·Identify opportunities to automate, standardize or simplify data engineering workflows ·Stay up to date with Databricks, Spark and cloud data engineering best practices Essential Skills & Experience ·Proven experience as a Data Engineer working in a modern cloud data platform ·Strong hands ...

Collaborate with senior developers and quantitative teams on new analytics capabilities Participate in debugging, testing, and production support activities Technology Environment Python Databricks/Apache Spark SQL Pandas/NumPy Git REST APIs Cloud technologies (Azure or AWS) Numerix (or equivalent pricing/risk platform) Jupyter/notebook … commercial Python development experience Strong understanding of data structures and software engineering fundamentals Experience working with data engineering or analytics tooling Exposure to Databricks, Spark, or cloud-based data platforms Understanding of SQL and relational databases Excellent problem-solving and communication skills STEM or Computer Science degree preferred Nice ...

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

experience working with data warehouses. Solid understanding of ETL/ELT design and data pipeline optimisation. Experience with big data frameworks such as Apache Spark, and exposure to streaming technologies like Kafka. Strong problem-solving skills and the ability to collaborate effectively across teams. What's on offer ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

focus on production-quality code • Solid understanding of ETL processes and best practices • Experience with SQL/NoSQL databases and big data tools (e.g. Spark, Pulsar) • Hands-on experience with cloud platforms (AWS, Azure, or GCP) • Experience developing and deploying data services end-to-end • Strong stakeholder communication ...

focus on production-quality code • Solid understanding of ETL processes and best practices • Experience with SQL/NoSQL databases and big data tools (e.g. Spark, Pulsar) • Hands-on experience with cloud platforms (AWS, Azure, or GCP) • Experience developing and deploying data services end-to-end • Strong stakeholder communication ...

capacity planning, skills development, and vendor contributions The individual: Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark. Experience with data modeling, metadata and lineage tooling; performance & cost optimization. Proven leadership in engineering teams; Agile/DevOps ways of working ...

such as Docker and orchestration tools like Kubernetes Strong problem-solving skills and attention to detail Desirable Skills Experience with big data technologies (e.g., Spark, Hadoop) Knowledge of MLOps practices and tools (e.g., MLflow, Kubeflow) Experience working with NLP, computer vision, or recommendation systems Understanding of data governance ...

code. LLMOps & AI Model Management: Familiarity with tools like MLFlow, LangChain, Hugging Face, Kubeflow, or similar platforms. Data Processing: Working knowledge of Databricks/Spark or comparable large-scale data processing tools. SQL: Solid capabilities in data querying and preparation. Data Architectures: Understanding of modern data infrastructure (lakehouses, data ...