126 to 150 of 151 Apache Spark Jobs in the UK

innovative thinking process and encourage it in others. Working within the agile framework at Chambers Skills and Experience Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation … design, build, and maintain scalable data pipelines for batch and streaming data using modern frameworks. Experience working with Databricks for big data processing, Spark-based transformations, and collaborative analytics workflows. Familiarity with Azure Data Factory (ADF) for data orchestration and integration across cloud environments. Skilled in version control using ...

Your new company Working for a renowned financial services organisation Your new role We're looking for a Senior Data Engineer to design and deliver scalable on prem, high-quality data solutions for low/ ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

/ELT) pulling from core platforms, cloud storage, databases and APIs Turn raw, messy, disparate data into curated, analysis-ready datasets Work with Python, Spark and SQL every day Develop models and schemas using a medallion architecture approach Help shape our data architecture with the Data Architect Implement governance … solve a problem you'll fit right in. What you'll bring You'll likely have: Hands-on expertise with Databricks Confidence in Python, Spark and SQL Experience building pipelines in Azure A solid understanding of data modelling and scalable architectures But just as important: You're inclusive ...

lead embedded data science teams, partnering effectively with senior stakeholders across multiple domains Hands-on technical expertise with tools such as Databricks, Python, Spark, and GCP/BigQuery Engineering mindset, with experience moving teams toward machine learning engineering best practices Credibility to lead long-tenured individual contributors while providing ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks ...

team engineering organisations Strong background in distributed systems, large-scale data pipelines and ML model deployment Deep understanding of modern data stacks such as Spark, Databricks or cloud-native architectures Solid understanding of modern frontend engineering, DevOps knowledge and experience delivering machine learning systems Strong communication, organisation and stakeholder ...

ensure A-Level grades are included on your CV) Basic scripting knowledge in Python or Bash Excellent customer-facing skills You have a sales spark - while this role isn't a focused sales role, this is required due to the nature of the role A motivated self-starter with ...

locally-hosted solutions. Technical Requirements Languages Expert-level Python and advanced SQL . Data Engineering Experience with ETL/ELT and orchestration tools like Apache Airflow or NiFi . On-Prem Tech Proficiency with Hadoop/HDFS , Spark , and containerization via Docker/Kubernetes (K3s/OpenShift) . ...

pairing, and sharing best practices. What You’ll Bring Technical Expertise: Proficiency in Python and SQL; experience with data processing frameworks such as Airflow, Spark, or TensorFlow. Data Engineering Fundamentals: Strong understanding of data architecture, data modelling, and scalable data solutions. You have hands-on experience deploying and running … systems in production, including model serving and monitoring. Cloud Platforms: Familiarity with AWS or Azure, including services like Apache Airflow, Terraform, or SageMaker. Data Quality Management: Experience with data versioning and quality assurance practices. You enjoy working in fast-moving environments where you can have real impact ...

practices like TDD, CI/CD, IaC and Observability. Skills: TypeScript/Node.js Backend APIs AWS or GCP Streaming systems experience (Flink, Kafka, Kinesis, Spark etc.) In return, you'll be given real ownership in an early-stage start-up environment where you can shape direction and genuinely make ...

/ELT pipelines integrating large, complex datasets Work with structured, semi-structured, and unstructured data across SQL and NoSQL technologies Develop solutions using Hadoop, Spark, and Splunk in large-scale environments Write maintainable Python code, applying object-oriented and functional programming principles Implement and maintain CI/CD pipelines … sustainability, and recoverability Essential Skills & Experience Proven experience as a Lead or Senior Data Engineer in enterprise-scale environments Hands-on expertise with Hadoop, Spark, and Splunk Advanced Python development skills Experience designing and optimising high-performance data pipelines Strong understanding of CI/CD, source control, and automated ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

data architectures Build scalable ETL workflows and data lake solutions using AWS-native services Engineer high-performance data processing solutions using Python, Java, Scala, Spark and SQL Integrate structured and unstructured data from multiple sources using APIs, scripting and cloud-native tooling Design and build robust AWS cloud data … expertise, particularly across data and analytics services Hands-on experience building ETL processes and data lake/warehouse architectures Proficiency in Python, Java, Scala, Spark and SQL Experience processing large-scale structured and unstructured datasets Experience extracting and integrating data via APIs and custom scripting Strong analytical thinking ...

Data Engineer Role: Data EngineerLocation: Newcastle Upon TyneSalary: TBC – Depending on experienceLevels: Senior Analyst, Specialist Hybrid Working: 3 days per week in our Newcastle, Cobalt business park office Please Note: Any offer of employment is ...

production-ready PySpark code. Strong understanding of data and data warehousing concepts, including: ETL/ELT, Data models, Dimensions and facts, Data marts, SCDs Spark Performance & Optimisation Strong knowledge of Spark execution concepts, including partitioning, optimisation, and performance tuning. Experience troubleshooting and improving distributed data processing pipelines. Python ...

Title: Lead Data Engineer - Hadoop, Spark, Pytthon Location: Sheffield - 3 days per week in the office Salary/Rate: Up to £617 per day inside IR35 Start Date: 02/03/2026 Job Type: Contract until November We are seeking a Lead Data Engineering Consultant with proven experience … leading and developing data engineering platforms. The ideal candidate will possess hands-on expertise in the following areas: Extensive enterprise experience with Hadoop, Spark, and Splunk. Proficiency in object-oriented and functional scripting, particularly in Python. Skilled in handling raw, structured, semi-structured, and unstructured data (SQL and NoSQL ...

work sits within a modern cloud data stack. You will work with technologies such as AWS, GCP or Azure, distributed processing frameworks like Spark, and modern data stores and warehouses such as Snowflake or BigQuery. The emphasis is on building robust, well structured data logic that scales alongside … looking for Strong, hands on experience using Python and SQL in production environments Experience building and maintaining data pipelines using technologies such as Spark, Snowflake, BigQuery, or similar Experience working in cloud environments such as AWS, GCP or Azure, with an understanding of data processing at scale Confidence working ...

Join our client in embarking on an ambitious data transformation journey using Databricks, guided by best practice data governance and architectural principles. To support this, we are recruiting for talented data engineers. As a major ...

with Azure Cosmos DB including query optimisation and throughput management Thorough understanding of Cosmos DB Change Data Feed (CDF) and integration and debugging of Spark with Cosmos CDF. High technical proficiency with Cosmos DB SDKs (e.g., .NET, Java, Python, Node.js) Thorough understanding of Spark distributed computing concepts. High … technical proficiency in PySpark and Spark Concepts. Experience with concurrency patterns, CLR, and scalable application design. Deep understanding of Azure services including Azure Functions, App Services, AKS, and Logic Apps. Experience with SQL API and familiarity with other APIs. Strong understanding of partitioning, indexing, and consistency levels. Experience with ...

with Azure Cosmos DB including query optimisation and throughput management Thorough understanding of Cosmos DB Change Data Feed (CDF) and integration and debugging of Spark with Cosmos CDF. High technical proficiency with Cosmos DB SDKs (e.g., .NET, Java, Python, Node.js) Thorough understanding of Spark distributed computing concepts. High … technical proficiency in PySpark and Spark Concepts. Experience with concurrency patterns, CLR, and scalable application design. Deep understanding of Azure services including Azure Functions, App Services, AKS, and Logic Apps. Experience with SQL API and familiarity with other APIs. Strong understanding of partitioning, indexing, and consistency levels. Experience with ...

mission-critical systems, driving technical excellence in data engineering and cloud-native development. The role requires strong hands-on expertise in Azure Cosmos DB, Spark, and distributed computing concepts. Key Responsibilities Develop cloud-native applications using Azure Cosmos DB (SQL API & Mongo API) Design and optimise NoSQL data models … Implement partitioning and indexing strategies for performance optimisation Optimise RU consumption and query performance Work with Cosmos DB Change Data Feed (CDF) and Spark integrations Develop reusable libraries using C#/.NET or Node.js Build CI/CD pipelines using GitLab Implement automated testing frameworks Collaborate with architecture ...

thinking to support our data-driven initiatives. Key Responsibilities: Lead the design, development, and optimisation of enterprise data platforms using tools such as Hadoop, Spark, and Splunk. Develop and maintain robust ETL/ELT data pipelines to handle raw, structured, semi-structured, and unstructured data (SQL and NoSQL). … ensuring service resilience and sustainability. Support continuous integration, delivery, and deployment (CI/CD) pipelines, incorporating source control practices. Qualifications: Extensive experience with Hadoop, Spark, and Splunk. Proficiency in Python scripting, with experience in handling various data formats. Strong background in building and optimising data pipelines and supporting ...

Head of Engineering - Remote - UK - £90,000-£130,000 GenAI, ML, Spark Why Join This Company? This is a leadership role for someone who wants to work in a truly AI-first engineering environment, not a business experimenting at the edges. The company is already well ahead … solving rather than legacy ways of working. You'll Work With Cross-functional teams spanning: Web Engineering (React/TypeScript) Data Engineering (distributed systems, Spark-based platforms) Data Science & Machine Learning A highly technical CTO who is deeply invested in AI adoption and modern engineering Product leadership following significant ...