76 to 100 of 117 Apache Spark Jobs in the UK

Kubernetes Strong Git-based development workflows Experience delivering features end-to-end in distributed teams Confident troubleshooting production issues in real time Desirable Skills Apache Spark (EMR) & Delta Lake PostgreSQL (Aurora) CI/CD using GitHub Actions IaC (Helm, CloudFormation) Monitoring tools (Sentry) Role Responsibilities … React frontend Build and maintain Dockerised microservices deployed to AWS (EKS) Troubleshoot and optimise performance across the stack Support data ingestion and transformation pipelines (Spark-based) Work cross-functionally with product and data specialists Help shape new features and influence engineering standards This role is 100% hands-on engineering ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

cloud data platforms Required Technical Expertise - Data Migration & Data Synchronisation Architecture - SQL/Python/Scala - Data Engineering & Data Pipelines - Relational & NoSQL Databases - Apache Spark - Databricks - Data Warehousing/Data Lakes - Cloud & On-Prem Data Integration Additional Requirements - Freelance contractor only - Must be eligible for security clearance - Availability ...

team Exposure to sectors such as defence, public sector and consumer/retail Familiarity with modern data and AI tooling such as Python, Spark, Databricks, Azure cloud, and Generative AI platforms (experience with specific tools is helpful but not essential) A mix of technical, product and operational experience – comfortable ...

team Exposure to sectors such as defence, public sector and consumer/retail Familiarity with modern data and AI tooling such as Python, Spark, Databricks, Azure cloud, and Generative AI platforms (experience with specific tools is helpful but not essential) A mix of technical, product and operational experience - comfortable ...

scripting skills in Python for automation, logging, and observability integrations. Expertise in SQL, especially Redshift, and in managing large-scale data processing frameworks like Spark and EMR. Hands-on experience with containerization (Docker, Kubernetes). Professional certifications such as AWS Certified Solutions Architect and AWS Certified DevOps Engineer … provisioning automation. Nice to Have Skills: Experience with Power BI, DAX, and data modeling for optimized reporting. Knowledge of big data orchestration tools like Apache Airflow. Previous exposure to regulated environments such as finance, healthcare, or gaming. Experience in a lottery or similar domain is a plus. Preferred Education ...

innovative thinking process and encourage it in others. Working within the agile framework at Chambers Skills and Experience Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation … design, build, and maintain scalable data pipelines for batch and streaming data using modern frameworks. Experience working with Databricks for big data processing, Spark-based transformations, and collaborative analytics workflows. Familiarity with Azure Data Factory (ADF) for data orchestration and integration across cloud environments. Skilled in version control using ...

e.g. Pandas, NumPy, Matplotlib) Data visualisation using Power BI or Tableau ideal Familiarity with Agile delivery methodologies Microsoft Fabric Power Apps, Power Automate, Dataverse Apache Spark/PySpark ...

experience with Python (2+ Years) Experience working with REST Microservices Strong SQL Experience working with very large data sets. Knowledge of big data tools (Spark, Kafka etc) Experience working in finance (Preferred) Strong formal education - ideally in Computer Science If this sounds of interest, then please do not hesitate ...

will be working across different client setups. Typically, that includes: AWS, Azure and GCP depending on the project Tools like Databricks, Snowflake and Synapse Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or Kinesis You will use your expertise to choose the right ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

deploy scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical … Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub ...

/ELT) pulling from core platforms, cloud storage, databases and APIs Turn raw, messy, disparate data into curated, analysis-ready datasets Work with Python, Spark and SQL every day Develop models and schemas using a medallion architecture approach Help shape our data architecture with the Data Architect Implement governance … solve a problem you'll fit right in. What you'll bring You'll likely have: Hands-on expertise with Databricks Confidence in Python, Spark and SQL Experience building pipelines in Azure A solid understanding of data modelling and scalable architectures But just as important: You're inclusive ...

services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory). Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability, and maintainability across cloud and on-prem environments. Champion data quality, governance, and observability, ensuring data is accurate, timely … Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub ...

services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory). Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability, and maintainability across cloud and on-prem environments. Champion data quality, governance, and observability, ensuring data is accurate, timely … Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark. Very useful - Python, SQL, JSON, LakeFS. Reasonable ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark. Very useful - Python, SQL, JSON, LakeFS. Reasonable ...

models for performance, scalability, and accuracy. Qualifications: Deep knowledge of neural networks (CNNs, RNNs, LSTMs, Transformers). Strong experience with data tools (Pandas, NumPy, Apache Spark). Solid understanding of NLP algorithms. Experience integrating ML models via RESTful APIs. Familiarity with CI/CD pipelines and deployment automation. ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks ...

team engineering organisations Strong background in distributed systems, large-scale data pipelines and ML model deployment Deep understanding of modern data stacks such as Spark, Databricks or cloud-native architectures Solid understanding of modern frontend engineering, DevOps knowledge and experience delivering machine learning systems Strong communication, organisation and stakeholder ...

locally-hosted solutions. Technical Requirements Languages Expert-level Python and advanced SQL . Data Engineering Experience with ETL/ELT and orchestration tools like Apache Airflow or NiFi . On-Prem Tech Proficiency with Hadoop/HDFS , Spark , and containerization via Docker/Kubernetes (K3s/OpenShift) . ...

technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale. This is a player-coach role: you will design … environment. Advanced hands-on expertise with: Python (including PySpark & data engineering patterns) AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks) MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices) Strong across broader AWS services: S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets ...

with EKS) Proficiency in Git-based development workflows Experience working independently and communicating effectively in distributed teams Desirable : Familiarity with (or happy to learn): Apache Spark (EMR) and Delta Lake PostgreSQL (Aurora) CI/CD (Github Actions) and IaC (Helm, CloudFormation) Monitoring (Sentry) Knowledge/interest in research … improve the TypeScript/React-based frontend application. Build and manage containerised services deployed on AWS, utilising Kubernetes orchestration. Support data and transformation workflows (Spark) Diagnose, troubleshoot, and improve overall application performance. Handle and resolve live production issues efficiently and under time pressure. Collaborate closely with product and data ...

/ELT pipelines integrating large, complex datasets Work with structured, semi-structured, and unstructured data across SQL and NoSQL technologies Develop solutions using Hadoop, Spark, and Splunk in large-scale environments Write maintainable Python code, applying object-oriented and functional programming principles Implement and maintain CI/CD pipelines … sustainability, and recoverability Essential Skills & Experience Proven experience as a Lead or Senior Data Engineer in enterprise-scale environments Hands-on expertise with Hadoop, Spark, and Splunk Advanced Python development skills Experience designing and optimising high-performance data pipelines Strong understanding of CI/CD, source control, and automated ...