24 of 24 Contract Apache Spark Jobs in England

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Warehouses. Design and oversee ingestion frameworks for enterprise data sources, including: Microsoft SQL Server (on-premises and Azure SQL) Oracle Databases Develop and optimize Apache Spark workloads using PySpark or Scala, ensuring performance, scalability, and operational efficiency. Assess existing or partially completed data platform implementations and provide remediation … integration with Databricks SQL Warehouses Experience integrating enterprise data sources such as: Microsoft SQL Server (on-premises and Azure SQL) Oracle Databases Strong Apache Spark expertise using PySpark and/or Scala, including performance tuning, partitioning strategies, and workload orchestration. Experience evaluating and remediating incomplete, underperforming, or legacy ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure as Code. Git/GitLab and modern software development practices. ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active SC Clearance ...

development experience Databricks and PySpark expertise Experience with cloud-based data platforms (AWS, Azure and/or GCP) Data Engineering and analytics at scale Apache Spark ecosystems Cyber Security or Security Analytics experience CI/CD and automated deployment practices Strong stakeholder engagement and communication skills You will ...

Senior Data Engineer (Scala/Spark) London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale … machine learning workflows at scale. As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services) Secondary skills API-based Data Integration & Microservices exposure Banking/Financial ...

Prometheus, ElasticSearch , Grafana, OpenTelemetry ) for metrics, logs and traces Platform Technologies & BSS Vendor Platforms - Open source workflow engines (Temporal) - Stream/batch processing (Flink, Spark) - Test automation: Robot Framework and BDD - can review BDD robot files - BSS vendor platforms: Salesforce, Amdocs, Ciena - BSS/OSS proficiency: awareness of Product ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services ...

/ELT design and implementation) Strong SQL & Data Modeling (query optimization, schema design) Programming Skills (Python/Scala for data processing) Big Data Processing (Spark/Databricks) Cloud Data Platforms (Azure/AWS/GCP - especially data services) Secondary s click apply for full job details ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation Knowledge of statistical methods and experimentation (A/B testing) preferred ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation Knowledge of statistical methods and experimentation (A/B testing) preferred ...

seasonality analysis) Solid foundation in statistical modelling and machine learning theory Proficiency in Python and experience working with large, complex datasets using SQL and Spark Ability to manage multiple priorities across a portfolio of projects Experience engaging both technical and business stakeholders Experience with libraries such as SciPy, Scikit ...

work on a Big Data Project in the finance sector. Required Experience: Java Big Data experience On-premises cluster experience Databases Kubernetes Kafka Apache Spark Preffered Experience: Finance sector experience ...

work on a Big Data Project in the finance sector. Required Experience: Java Big Data experience On-premises cluster experience Databases Kubernetes Kafka Apache Spark Preffered Experience: Finance sector experience ...

Analytics Engineer London, hybrid 3 times a week Department: IT Type: 3-month rolling contract Outside IR35 Position: Analytics Engineer The organisation is seeking an experienced Analytics Engineer to join its Data & Analytics team. The ...

delivery of secure, scalable and high-performance data solutions within a regulated environment. This role will focus heavily on the Elastic Stack and Apache NiFi, supporting enterprise-scale data ingestion, transformation and analytics capabilities across complex secure systems. The successful candidate will play a key role in architecting resilient … compliance with strict governance and security standards. KEY RESPONSIBILITIES Data Architecture & Engineering - Design and implement scalable, resilient data pipeline architectures using Elastic Stack and Apache NiFi - Develop secure frameworks for data ingestion, transformation and integration - Build and maintain batch and real-time data processing pipelines - Ensure high standards ...

workloads. Practical experience working with AWS cloud services, including compute, storage, and security tooling. Infrastructure-as-code experience using Terraform. Distributed data processing using Apache Spark. Workflow scheduling and orchestration with Apache Airflow. Containerisation and environment management using Docker. CI/CD pipeline implementation and maintenance using GitLab. ...

Senior Software Engineer - Real-Time Data (Kafka/Spark) Chester | Hybrid working | 12-month contract | Inside IR35 | Global Investment Bank A leading global investment bank is looking for a hands-on Senior Engineer to build high-performance, real-time data platforms supporting regulatory compliance across global operations. This … processing , ideal for someone who enjoys solving complex data challenges at scale. What You'll Do Design and build Kafka-based streaming pipelines and Spark processing jobs Develop low-latency, high throughput systems handling large-scale datasets Create data matching and reconciliation solutions across multiple sources Work on event ...