1 to 25 of 39 Apache Spark Jobs in the UK excluding London

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

evolution of data architecture and help shape the organisations data engineering capabilities. Key Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark and Python. Develop and maintain robust ELT processes to ingest, transform and integrate data from multiple sources. Contribute to the design and evolution … platform performance. Essential Skills & Experience Strong hands-on experience with Databricks . Proven experience designing, implementing and optimising data pipelines. Strong knowledge of Apache Spark and Python . Understanding of Lakehouse architecture and modern data engineering principles. At least five years experience across ELT, data engineering and data ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands-on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud-native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

Data Engineer 6 Month Contract Location: Hereford (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

programming skills, with experience developing and maintaining data pipelines Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies Knowledge of data visualisation tools ...

Python and SQL; APIs; test automation; data structures; distributed processing; Git; CI/CD; containers; infrastructure awareness; production debugging and performance optimization. Data engineering: Spark or Databricks, Kafka or equivalent streaming, ETL/ELT, batch and real-time pipelines, data contracts, lineage, cataloguing, quality rules, and scalable cloud storage. ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP. Container Orchestration: Experience packaging and deploying workloads with Helm ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

approaches Experience with Generative AI techniques such as retrieval-augmented generation, fine-tuning, or prompt engineering Exposure to distributed data processing frameworks, such as Spark Exposure to fraud detection, anti-money laundering, anomaly detection, or graph and network-based modelling Awareness of model risk management, governance, and regulatory controls ...

Proficiency in the Python data science ecosystem (e.g. NumPy, Pandas, scikit-learn, PyTorch, TensorFlow). Experience with big data and distributed analytics platforms (e.g. Spark, Hadoop). Hands-on experience with Natural Language Processing, graph analytics and advanced analytical methods. Experience using cloud analytics platforms, with strong ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

effectively with technical teams including Data Scientists and MLOps Engineers. Desirable Skills Experience working with Data Science and/or MLOps environments. Experience with Apache Spark. Experience with Scala. Experience working with S3 and cloud storage technologies. Experience with graph databases. Experience with Elasticsearch. Experience with vector databases/ ...

model validation. Experiment tracking and reproducibility tools, for example MLflow, Weights & Biases or equivalent. Data wrangling and analysis using Pandas, NumPy, SQL; familiarity with Spark or similar is a plus. Model optimisation and research-to-prototype deployment fundamentals, including ONNX, TorchScript, FastAPI/gRPC and GPU acceleration, with CUDA ...

robust data models, ETL pipelines, metadata frameworks, and modern data warehousing solutions. Design and optimise large-scale streaming data platforms using technologies such as Apache Flink and other real-time processing frameworks. Ensure all solutions comply with industry best practices, governance frameworks, and regulatory requirements, maintaining the highest standards … Lambda, Redshift , and related data services. Proven experience designing secure architectures within sensitive or highly regulated environments. Hands-on experience with Apache Flink and real-time data processing architectures. Strong understanding of data modelling methodologies, architectural patterns, and industry standards. Expertise across ETL processes, data integration, data warehousing, distributed ...