26 to 50 of 61 Apache Spark Jobs in the UK excluding London

engineering across ingestion, transformation and data centralisation. Experience with tools such as Glue, Fivetran, Kafka, Dataflow, ADF or Kinesis. Transformation experience using dbt, Spark or Databricks. Confidence working directly with clients and engaging in technical discussions. Exposure to dashboards, analytics, ML or LLM concepts is beneficial, but not essential. ...

engineering across ingestion, transformation and data centralisation. Experience with tools such as Glue, Fivetran, Kafka, Dataflow, ADF or Kinesis. Transformation experience using dbt, Spark or Databricks. Confidence working directly with clients and engaging in technical discussions. Exposure to dashboards, analytics, ML or LLM concepts is beneficial, but not essential. ...

engineering across ingestion, transformation and data centralisation. Experience with tools such as Glue, Fivetran, Kafka, Dataflow, ADF or Kinesis. Transformation experience using dbt, Spark or Databricks. Confidence working directly with clients and engaging in technical discussions. Exposure to dashboards, analytics, ML or LLM concepts is beneficial, but not essential. ...

Ensure high standards of data quality, governance, security and accessibility across data platforms Develop scalable data transformation logic using Python, SQL and Spark-based frameworks Monitor pipeline performance and troubleshoot data workflows to ensure reliability and efficiency Work closely with data scientists, analysts and engineering teams to support advanced … data services (such as S3, Glue, Redshift, EMR, Athena, Lambda or similar) Strong programming experience with Python and SQL Hands-on experience with Spark or distributed data processing frameworks Proven experience building and maintaining data pipelines, data lakes and data warehouses Strong understanding of data modelling, metadata management ...

Kafka Exposure to cloud platforms (AWS, Azure, or GCP) Familiarity with containerisation technologies (Docker, Kubernetes) Experience with distributed data processing tools such as Spark or Flink Additional Information Daily rate: £850 - £900 Hybrid working model in London Initial contract with strong likelihood of extension McGregor Boyall is an equal ...

common ML frameworks such as PyTorch, TensorFlow, or scikit-learn * Exposure to ML/MLOps tooling such as SageMaker, MLflow, or TFServing * Experience with Spark, Databricks, CI/CD tools, and streaming or orchestration systems like Kafka or Airflow How to Apply If you are interested in this Machine ...

databases into ADLS Gen2 and OneLake Develop structured data models aligned to Lakehouse, Medallion Architecture, and Delta Lake patterns Continuously profile, debug, and optimise Spark jobs, SQL queries, and Python scripts for maximum performance and cost-efficiency at scale Champion DevOps best practices: implement infrastructure-as-code (Terraform), automated … Essential Technical Skills Advanced proficiency in Python and PySpark, writing clean, modular, object-oriented code for data transformations Strong command of SQL (T-SQL, Spark SQL) for data exploration, validation, and final-stage modelling Deep hands-on experience with Microsoft Fabric and its tooling such as Azure Data Factory ...

Experience with cloud platforms (AWS) and containerisation (such as Docker, Podman, Kubernetes) Experience working in secure or regulated environments Exposure to big data technologies (Spark, Hadoop) and Apache tools Familiarity with Agile and DevOps practices Industry certifications (e.g., TensorFlow Developer, AWS Machine Learning Specialty) Knowledge of NLP, computer ...

influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing … Engineer or Solutions Architect certification (desirable). Databricks Certified Data Engineer or Machine Learning Associate (desirable). Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable). Knowledge of machine learning and AI on Databricks (desirable). ...

Kubernetes Strong Git-based development workflows Experience delivering features end-to-end in distributed teams Confident troubleshooting production issues in real time Desirable Skills Apache Spark (EMR) & Delta Lake PostgreSQL (Aurora) CI/CD using GitHub Actions IaC (Helm, CloudFormation) Monitoring tools (Sentry) Role Responsibilities … React frontend Build and maintain Dockerised microservices deployed to AWS (EKS) Troubleshoot and optimise performance across the stack Support data ingestion and transformation pipelines (Spark-based) Work cross-functionally with product and data specialists Help shape new features and influence engineering standards This role is 100% hands-on engineering ...

team Exposure to sectors such as defence, public sector and consumer/retail Familiarity with modern data and AI tooling such as Python, Spark, Databricks, Azure cloud, and Generative AI platforms (experience with specific tools is helpful but not essential) A mix of technical, product and operational experience - comfortable ...

team Exposure to sectors such as defence, public sector and consumer/retail Familiarity with modern data and AI tooling such as Python, Spark, Databricks, Azure cloud, and Generative AI platforms (experience with specific tools is helpful but not essential) A mix of technical, product and operational experience - comfortable ...

ongoing basis. Contribute to the planning, research, recommendation, and implementation of current and future IT strategies. PERSON SPECIFICATION: Essential Demonstrable experience of Spark (PySpark) and SQL. Demonstrable experience of Fabric components. Demonstrable experience of working on enterprise environments. Experience of designing architectural Fabric processes. Experience of Azure Data Factory. ...

innovative thinking process and encourage it in others. Working within the agile framework at Chambers Skills and Experience Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation … design, build, and maintain scalable data pipelines for batch and streaming data using modern frameworks. Experience working with Databricks for big data processing, Spark-based transformations, and collaborative analytics workflows. Familiarity with Azure Data Factory (ADF) for data orchestration and integration across cloud environments. Skilled in version control using ...

like ER/Studio, ERwin, or Sparx EA (preferred). Experience reverse engineering models from existing databases. Proficiency in distributed data processing frameworks (e.g., Spark, Flink, Hadoop). Experience with API management and gateway tools and services. Experience of RESTful APIs for ingesting and exposing data. Proven track record … with data governance, quality, lineage, and security practices. Experience with real-time/streaming data technologies, data Ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub). Experience in, or knowledge of, DevSecOps Tooling and Processes. Self-starter with the ability to appropriately prioritise ...

will be working across different client setups. Typically, that includes: AWS, Azure and GCP depending on the project Tools like Databricks, Snowflake and Synapse Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or Kinesis You will use your expertise to choose the right ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

/ELT) pulling from core platforms, cloud storage, databases and APIs Turn raw, messy, disparate data into curated, analysis-ready datasets Work with Python, Spark and SQL every day Develop models and schemas using a medallion architecture approach Help shape our data architecture with the Data Architect Implement governance … solve a problem you'll fit right in. What you'll bring You'll likely have: Hands-on expertise with Databricks Confidence in Python, Spark and SQL Experience building pipelines in Azure A solid understanding of data modelling and scalable architectures But just as important: You're inclusive ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark. Very useful - Python, SQL, JSON, LakeFS. Reasonable ...

experience building APIs at scale ⚙️ You’ll need at least 2 of the below: ☁️ AWS (3+ years) – S3, Lambda, DynamoDB 📊 Big Data (3+ years) – Spark, Snowflake, Kafka 📦 Containers (3+ years) – Kubernetes/OpenShift 💳 Payments experience (2–3 years) – card payments/fintech/transaction systems 👉 We’re being strict ...

candidates with experience of one or more of the major cloud platforms. AWS, Azure and GCP depending on the project Databricks, Snowflake and Synapse Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or Kinesis You're expected to pick the right tools ...

technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale. This is a player-coach role: you will design … environment. Advanced hands-on expertise with: Python (including PySpark & data engineering patterns) AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices) Strong across broader AWS services: S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets ...

technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale. This is a player-coach role: you will design … environment. Advanced hands-on expertise with: Python (including PySpark & data engineering patterns) AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices) Strong across broader AWS services: S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets ...

with EKS) Proficiency in Git-based development workflows Experience working independently and communicating effectively in distributed teams Desirable : Familiarity with (or happy to learn): Apache Spark (EMR) and Delta Lake PostgreSQL (Aurora) CI/CD (Github Actions) and IaC (Helm, CloudFormation) Monitoring (Sentry) Knowledge/interest in research … improve the TypeScript/React-based frontend application. Build and manage containerised services deployed on AWS, utilising Kubernetes orchestration. Support data and transformation workflows (Spark) Diagnose, troubleshoot, and improve overall application performance. Handle and resolve live production issues efficiently and under time pressure. Collaborate closely with product and data ...