1 to 25 of 65 Remote Apache Spark Jobs

role We are seeking a high‐skilled Senior Software Engineer with deep expertise in API Development using Spring Boot while also possessing knowledge of Apache Spark‐based Big Data pipelines using Airflow. This hybrid position bridges software engineering and data engineering, requiring the ability to design, build … teams to architect and manage robust, production‐grade data products that power critical analytics and measurement capabilities. You will work with technologies such as Apache Spark, Apache Airflow, Trino, Druid, Spring Boot, StarRocks, and Looker to deliver reliable, high‐performance solutions. The ideal candidate is a proactive ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Data Engineer Salary up to £70,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via ABOUT THE CLIENT Due to continued growth, we're seeking a skilled Data Engineer … optimise scalable data pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems, whilst … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

engineering, data science, or a hybrid role Strong programming skills in Python and SQL Experience building data pipelines and working with large datasets (e.g., Spark, Airflow, or similar tools) Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

enterprise data strategies, optimize pipelines (ETL/ELT), and incorporate technologies such as cloud platforms (e.g., AWS GovCloud, Azure Government), Big Data tools (e.g., Spark, Hadoop), and relational/NoSQL databases for secure, performant solutions. Designs approaches for automated data labeling, lifecycle management, monitoring, real-time ingestion, and integration ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

skills Experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or similar Hands-on experience with transformation frameworks such as dbt or Spark Experience with cloud data warehouses such as Snowflake, BigQuery, Redshift, or Databricks Strong understanding of data modeling principles across analytical and operational systems Experience with ...

/or Extract/Load/Transform (ELT) tools, including both batch and real-time data transmission applications such as SSIS, Informatica, Kafka, Spark, MuleSoft, or equivalent softwareAdvanced knowledge of database technologies such as Microsoft SQL Server, Oracle, or NoSQL databasesAdvanced knowledge of multiple programming languages, such as Python ...

/or Extract/Load/Transform (ELT) tools, including both batch and real-time data transmission applications such as SSIS, Informatica, Kafka, Spark, MuleSoft, or equivalent softwareAdvanced knowledge of database technologies such as Microsoft SQL Server, Oracle, or NoSQL databasesAdvanced knowledge of multiple programming languages, such as Python ...

/or Extract/Load/Transform (ELT) tools, including both batch and real-time data transmission applications such as SSIS, Informatica, Kafka, Spark, MuleSoft, or equivalent softwareAdvanced knowledge of database technologies such as Microsoft SQL Server, Oracle, or NoSQL databasesAdvanced knowledge of multiple programming languages, such as Python ...

handling and analysis skills Understanding of digital ways of working and collaboration tools Strong proficiency in Python using industry‐standard libraries. Solid experience with Spark and PySpark for distributed data processing. Strong SQL skills, including performance‐focused query design. Hands‐on experience with Databricks on Azure, including platform usage ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

through influence with strong collaborator engagement skills. Preferred Experience Digital Twin, asset virtualization, or industrial data platforms Graph technologies (e.g., Neo4j), big data ecosystems (Spark, Hive, Hadoop) Databricks, Python, and modern analytics platforms Delivery of data products at enterprise scale Key Competencies Strategic product thinking with strong execution bias ...

Senior Data Engineer (Scala/Spark) London | Hybrid Working | Initial 12 month contract | £450-550pd PAYE + holiday pay We are partnering with a world-leading technology organisation seeking an experienced Senior Data Engineer to join a high-performing engineering team responsible for building and operating large-scale … machine learning workflows at scale. As a Senior Data Engineer, you will design, build and operate large-scale batch data pipelines using Scala and Apache Spark. Working alongside Machine Learning Engineers, Researchers and Platform Engineers, you will help develop the infrastructure and tooling required to process vast datasets ...