1 to 25 of 38 Remote Apache Spark Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Software Developer Permanent | Hybrid (willing to travel to Newcastle) | Python | Apache | Data | SC cleared Hybrid work arrangement. Office attendance is up to 60%. Location is flexible: London, Leeds, Newcastle, Sheffield, Blackpool, Manchester and Birmingham. You must be willing to travel to Newcastle when required. The Role … looking for Software Developers with strong Python and data processing with Apache Spark responsible to lead the design, development, and operation of data-driven applications and pipelines in a collaborative Dev and DevOps environment. The role focuses on writing and improving application code while working closely with DevOps ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

equivalent experience. Strong programming skills (Python, SQL, R, or similar). Experience with cloud data platforms (Azure, AWS, GCP) and big data technologies (Spark, Hadoop). Knowledge or experience with Denodo is an advantage. Proficiency in BI and data visualisation tools (Power BI, Tableau). Solid understanding of data ...

streaming (e.g., Kafka), and cloud services (e.g., AWS, GCP, Azure). Programming Skills: Programming skills in languages like Python (NumPy, pandas, PySpark), Java (Core Spark with Java, functional interface, collections), or Scala with experience in automation and scripting. Infrastructure Tools: Experience with containerization and orchestration tools like Docker ...

Strong programming experience in Java (preferred) or Python Experience building ETL/ELT or streaming data pipelines Hands-on experience with Kafka, Flink, or Spark (streaming experience preferred) Good understanding of stream-processing concepts (event time, state, backpressure) Experience with cloud platforms (AWS preferred; Azure or GCP also considered ...

lead quality assurance and testing activities within a large-scale cloud data engineering programme operating across a modern AWS-native technology ecosystem, including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, DynamoDB, and related cloud services. This role combines hands-on technical leadership with test strategy ownership . … reconciliation, and performance under load Oversee PySpark-based data validation and interrogation activities on AWS EMR Support testing and monitoring of workflows orchestrated through Apache Airflow Drive root cause analysis for production data issues and platform defects Risk Management & Reporting Identify, assess, and communicate quality risks across releases ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

with client-facing delivery as part of an agile and collaborative environment. Responsibilities Design, develop and optimize end-to-end data pipelines using Databricks, Spark/PySpark and Python Lead the full pipeline lifecycle: analysis, architecture, development, testing, deployment and monitoring Implement ingestion, normalization and standardization workflows for high ...

with cross-functional teams to deliver data-driven solutions Support data modelling, governance, and quality initiatives What were looking for Experience with Python, SQL, Spark (and ideally Scala) Hands-on delivery of ML solutions in production Exposure to cloud platforms (AWS, Azure, or GCP) Understanding of DevOps, containers ...

such as Docker and orchestration tools like Kubernetes Strong problem-solving skills and attention to detail Desirable Skills Experience with big data technologies (e.g., Spark, Hadoop) Knowledge of MLOps practices and tools (e.g., MLflow, Kubeflow) Experience working with NLP, computer vision, or recommendation systems Understanding of data governance ...

batch data processing systems for voice compliance assurance Low‐latency record matching and reconciliation engines handling billions of events Scalable data services operating across Spark, Kafka, Hive and Hadoop Production‐grade pipelines supporting regulatory evidence, audit and controls Foundations for near‐real‐time compliance signal generation across global voice … platforms Key Responsibilities Design and develop high‐performance distributed systems for large‐scale voice data processing Build and optimise Spark‐based processing jobs for high‐volume and high‐frequency workloads Engineer Kafka‐based streaming pipelines with strong delivery guarantees and low end‐to‐end latency Develop robust data matching ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

Strong analytical and problem-solving skills Excellent communication and stakeholder engagement skills Technology Stack Python, SQL, Databricks, PyTorch, Transformers/LLMs, Prompt Engineering, Pandas, Spark, AWS, Azure, PostgreSQL, FastAPI, Git, CI/CD, Knowledge Graphs What's on Offer Fully remote UK-based role Opportunity to work on cutting ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions are scalable ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

translate business needs into technical roadmaps What were looking for Strong experience in data architecture & large-scale systems design Expertise across Python, SQL, Spark, Scala Deep knowledge of AI/ML frameworks and modern data platforms Experience with cloud-native architecture and DevOps (Kubernetes, Terraform, CI/CD) Proven ...

experience with Generative AI and LLMs, including prompt engineering, fine-tuning, evaluation, and production deployment. Strong command of modern data stacks, including Python, SQL, Spark, BigQuery, and Snowflake. Clear communication skills, able to explain complex AI concepts to both technical and non-technical audiences. Benefits and perks There ...

experience with Generative AI and LLMs, including prompt engineering, fine‐tuning, evaluation and production deployment. Strong command of modern data stacks, including Python, SQL, Spark, BigQuery and Snowflake. Clear communication skills, able to explain complex AI concepts to both technical and non‐technical audiences. Benefits and perks ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...