1 to 25 of 46 Remote Apache Spark Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

role We are seeking a high‐skilled Senior Software Engineer with deep expertise in API Development using Spring Boot while also possessing knowledge of Apache Spark‐based Big Data pipelines using Airflow. This hybrid position bridges software engineering and data engineering, requiring the ability to design, build … teams to architect and manage robust, production‐grade data products that power critical analytics and measurement capabilities. You will work with technologies such as Apache Spark, Apache Airflow, Trino, Druid, Spring Boot, StarRocks, and Looker to deliver reliable, high‐performance solutions. The ideal candidate is a proactive ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

data estate. Deep expertise in Microsoft Azure cloud services — ADF, ADLS, Synapse, Purview. Databricks — Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large‐scale distributed computing and performance tuning. Microsoft Fabric — OneLake and DirectLake integration. Azure Synapse Analytics for enterprise‐scale data warehousing. Infrastructure ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

Extensive expertise in architectural design and oversight of both distributed and non-distributed compute architectures, with deep proficiency in languages such as Python, SQL, Spark, and Scala. Deep understanding of architectural implementations for intelligent solutions (Machine Learning, Agentic AI etc). Experience architecting CI/CD and MLOps frameworks ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

engineering, data science, or a hybrid role Strong programming skills in Python and SQL Experience building data pipelines and working with large datasets (e.g., Spark, Airflow, or similar tools) Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

such as Docker and orchestration tools like Kubernetes Strong problem-solving skills and attention to detail Desirable Skills Experience with big data technologies (e.g., Spark, Hadoop) Knowledge of MLOps practices and tools (e.g., MLflow, Kubeflow) Experience working with NLP, computer vision, or recommendation systems Understanding of data governance ...

batch data processing systems for voice compliance assurance Low‐latency record matching and reconciliation engines handling billions of events Scalable data services operating across Spark, Kafka, Hive and Hadoop Production‐grade pipelines supporting regulatory evidence, audit and controls Foundations for near‐real‐time compliance signal generation across global voice … platforms Key Responsibilities Design and develop high‐performance distributed systems for large‐scale voice data processing Build and optimise Spark‐based processing jobs for high‐volume and high‐frequency workloads Engineer Kafka‐based streaming pipelines with strong delivery guarantees and low end‐to‐end latency Develop robust data matching ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

Strong analytical and problem-solving skills Excellent communication and stakeholder engagement skills Technology Stack Python, SQL, Databricks, PyTorch, Transformers/LLMs, Prompt Engineering, Pandas, Spark, AWS, Azure, PostgreSQL, FastAPI, Git, CI/CD, Knowledge Graphs What's on Offer Fully remote UK-based role Opportunity to work on cutting ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

scaling data platforms, with a strong focus on Azure Cloud technologies (e.g., Azure Data Factory, Azure Synapse, Azure Data Lake, Azure SQL Database, Databricks, Apache Spark). Proven experience leading the implementation of data platform strategies and managing large‐scale data infrastructure initiatives using Azure. Demonstrated experience implementing ...

with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions are scalable ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...