24 of 24 Remote/Hybrid Apache Spark Jobs in the UK excluding London

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

design, build and operate reliable, scalable data pipelines and lakehouse solutions that enable analytics, AI and GenAI. You will work hands-on with Databricks, Apache Spark and the Azure data ecosystem to ingest, transform and serve trusted data products, taking ownership from development through to production support … Databricks focus, you will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage Databricks, Apache Spark and Azure services to develop and optimise batch and streaming pipelines, implement Delta Lake lakehouse patterns, and deliver well-governed data sets that ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

focus, you will be an integral part of our team dedicated to building scalable and secure data platforms. You will combine deep Databricks and Apache Spark expertise with strong client engagement skills to shape solutions from early-stage opportunity through delivery. You will lead our Databricks partnership activity … sales by contributing to discovery, solution design, estimates, and compelling client-facing materials. Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data across cloud and on-premises sources and targets. Design and oversee the delivery of secure data warehouses ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

computer science, IT or a related field, or in bioinformatics with a demonstrated IT expertise Experience in big data technologies and frameworks, such as Apache Spark, Hadoop or similar platforms Hands-on experience with CI/CD (GitLab CI/GitHub Actions) Familiarity with Java Familiarity with Google ...

About the Job you are considering: Results-driven Hadoop Spark Data Engineer with strong expertise in designing and implementing scalable big data solutions using Scala and Apache Spark. Experienced in working with the Hadoop ecosystem, including HDFS, Hive, and YARN, to process and analyze large datasets efficiently. Skilled … Azure. Your Skills: Proficient in Scala programming with strong expertise in functional programming concepts for building scalable data applications. Extensive experience in Apache Spark (Core, SQL, and Streaming) for processing large-scale distributed data efficiently Strong knowledge of Hadoop ecosystem components including HDFS, YARN, Hive, and HBase. Skilled ...

version control. Data Engineering Stack Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools ...

native solutions that support analytics, automation, and digital transformation. Data Engineering, Full Stack & Platform Development Design and implement scalable data pipelines using tools like Apache Spark, Airflow, or dbt.Build and maintain data lakes, warehouses, and real-time streaming solutions. Develop APIs and microservices to expose data securely ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

through influence with strong collaborator engagement skills. Preferred Experience Digital Twin, asset virtualization, or industrial data platforms Graph technologies (e.g., Neo4j), big data ecosystems (Spark, Hive, Hadoop) Databricks, Python, and modern analytics platforms Delivery of data products at enterprise scale Key Competencies Strategic product thinking with strong execution bias ...

Senior Data Engineer (Scala/Spark) London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale … machine learning workflows at scale. As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets ...

data modelling and processing but focuses on practical ability to implement solutions using a range of technologies and languages including SQL, Python, Databricks, Apache Spark, Lakehouse and various RDBMS. This is a senior, hands-on role. You will develop and support more junior team members whilst working ...

Education Required Approximately 10 years of industry experience implementing data integration tools such as Microsoft SSIS, Azure Data Factory, Databricks, Glue, Step Functions, Airflow, Apache Flume/Sqoop/Pig, and similar technologies. 3–5 years of team management experience. 3–5 years of consulting experience (preferred). Minimum … knowledge of data warehousing, OLTP systems, data integration, and the software development lifecycle (SDLC). Hands-on experience with big data frameworks such as Spark, Hadoop, or Hive, including derivatives like PySpark (preferred), Spark Scala, or Spark SQL, along with familiarity with development-accelerating libraries and frameworks. ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

Person We Are Looking For Proven experience delivering large-scale Databricks transformation projects Strong Financial Services or enterprise consulting background Deep expertise across Azure, Spark, Delta Lake and modern data engineering practices Experience building and leading high-performing technical teams Strong commercial awareness and stakeholder management experience Ability ...

using data mining techniques and with statistical rigor Strong technical skills regarding data analysis, statistics, machine learning and programming. Strong working knowledge of Hadoop, Spark, SQL, Python and/or R. Excellent communication skills Strong skills in translating business problems into data problems Strong data visualization skills Strong skills ...

insights that drive the AI product roadmap forward. Tech Stack Includes : AWS: EC2, Athena, S3, Kinesis, Lambda, Step Functions, RDS, QuickSight Languages & Frameworks: Python, Spark, TensorFlow, LLMs & LLM coding tools Databases & Search: MySQL, PostgreSQL, SOLR Requirements Experience: Minimum 3 years of post-degree commercial experience in a data science ...

platform role Strong hands-on experience with AWS (EC2, S3, RDS, Lambda, Redshift, Glue, Athena) Experience with real-time and batch data processing - Apache Flink, Kafka, Spark, or similar Proficiency in one or more of: Python, Java, Scala Experience with data modelling , ETL processes, and distributed systems Comfortable ...

need to potentially work in either Edinburgh or Glasgow depending on client expectations. What You’ll Do Design and develop robust pipelines using DeltaLake, Spark Structured Streaming, and Unity Catalog Build real-time event-driven solutions with tools such as Kafka and Azure Event Hubs Apply DevOps principles … knowledge of DevOps tools and CI/CD processes Bonus Points For Development experience in Scala or Java Familiarity with Cloudera, Hadoop, HIVE, and Spark ecosystem Understanding of data privacy regulations, including GDPR, and experience working with sensitive data Ability to learn and adapt new technologies quickly to meet ...

Senior Data Engineer 6 Month contract with extension Full-time Inside IR35 Hybrid 1 day every two weeks in the office Nottingham - NG1 4BX Join our client in embarking on an ambitious data transformation journey ...

Senior Software Engineer - Real-Time Data (Kafka/Spark) Chester | Hybrid working | 12-month contract | Inside IR35 | Global Investment Bank A leading global investment bank is looking for a hands-on Senior Engineer to build high-performance, real-time data platforms supporting regulatory compliance across global operations. This … processing , ideal for someone who enjoys solving complex data challenges at scale. What You'll Do Design and build Kafka-based streaming pipelines and Spark processing jobs Develop low-latency, high throughput systems handling large-scale datasets Create data matching and reconciliation solutions across multiple sources Work on event ...