25 of 25 Remote/Hybrid Apache Spark Jobs in the North of England

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

replay, backfill, recoverability and schema evolution Define and enhance operational telemetry, monitoring and platform observability Streaming & Distributed Data Processing Build and optimise solutions using:Apache Kafka Apache NiFi Apache Spark/PySpark Hadoop ecosystem technologies Hive and large-scale data storage platforms Develop scalable matching, enrichment … interested in engineers who can demonstrate strong expertise across several of the following areas: Core Technical Skills Python development in enterprise environments Apache Spark/PySpark Apache Kafka Apache NiFi Hadoop and Hive Advanced SQL REST API integration and development Data modelling and data lineage Distributed ...

engineers where needed. Core Data Engineering Strong programming proficiency in Java (preferred) or Python. Hands-on experience with at least one of: Kafka, Flink, Spark (Flink/Kafka preferred for streaming). Solid understanding of stream processing concepts (e.g., event time, state, backpressure). Understanding of software engineering best ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

solutions are secure, scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines to ensure trusted ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We … version control.* **Data Engineering Stack** Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL.* **Orchestration** Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect.* **Data Quality and Lineage** Familiarity with tools ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like ...

programming skills, with experience developing and maintaining data pipelines Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies Knowledge of data visualisation tools ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

Lambda, ECS, RDS, S3, SQS, API Gateway, and CloudWatch. Design, develop, and maintain data pipelines and ETL workflows using technologies such as Glue, Spark/PySpark, Kafka, or similar platforms. Implement and maintain Infrastructure as Code (IaC) using Terraform or equivalent tools. Drive observability, monitoring, logging, and operational readiness ...

into business processes and decision-making.**Technical Skills:**Hands-on experience in one or more of the following areas:Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analyticsCloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCPAI Technologies – GenAI frameworks, Agentic AI, vector databases, semantic search ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

modern cloud engineering practices. Wherever you land, you’ll be working with some of the biggest datasets in the UK — using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. Cloud-native AI engineering and large-scale data platforms, leveraging Google Cloud technologies and modern ...

modern cloud engineering practices. Wherever you land, you'll be working with some of the biggest datasets in the UK - using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Cloud-native AI engineering and large-scale data platforms ...

support comprehensive data analysis and reporting. Implement and uphold best practices in data governance, security, and compliance, using tools like Azure Purview, UnityCatalog, and Apache Atlas tomaintaindata integrity and trust. Ensure data quality and integrity through meticulous attention to detail and rigorous QA processes, continually refining and optimising data … ease. Understanding of Data Lakehouse Architecture:You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions:Youpossessexperience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or evenon-premiseSpark clusters, usingPySparkor Spark SQL to manage ...

Nice to have: Proven experience leading projects and managing teams, Experience in presales, client workshops, or building Proof-of-Concepts (PoCs), Familiarity with Hadoop, Spark, Research or R&D background. Soft Skills Ability to structure projects teams, make technical decisions, and own outcomes, Ability to manage multiple projects ...

Python and SQL; APIs; test automation; data structures; distributed processing; Git; CI/CD; containers; infrastructure awareness; production debugging and performance optimization. Data engineering: Spark or Databricks, Kafka or equivalent streaming, ETL/ELT, batch and real-time pipelines, data contracts, lineage, cataloguing, quality rules, and scalable cloud storage. ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with … platforms such as Amazon Bedrock. Experience developing data-based apps in Streamlit/Shiny Skills in ‘Big Data’ technology tools like Hadoop/Spark Knowledge of Infrastructure as Code tools like CloudFormation/Terraform If you can bring some of these skills and experiences, along with transferable strengths ...

Senior Data Engineer – SQL/Python/AWS Adria Solutions is recruiting a Senior Data Engineer for a growing business developing a new digital services platform. With established backing, the business is building its data ...

deploying scalable data platforms end-to-end on AWS and/or Databricks. Production Coding: Strong, hands-on production experience with Python, SQL, Spark, and Databricks or AWS data tools. Consulting/Client Delivery: Experience working in a consultancy, systems integrator, or client-facing setting with direct accountability …/or AWS certifications at Associate or Professional level. Streaming Architectures: Practical experience with real-time or event-driven streaming (Kafka/MSK, Kinesis, Spark Structured Streaming). Advanced Data Architectures: Exposure to Data Mesh concepts, data products, or federated data governance models. AI/ML Infrastructure: Exposure ...

team capability growth and development. What We’re Looking For Experience as a Data Engineer with strong skills in areas such as Python, NumPy, Spark and Apache Pig. Proven experience in designing, developing and deploying scalable systems. Familiarity with CI/CD, containerisation, deployment technologies and cloud platforms ...