1 to 25 of 76 Apache Spark Jobs in London

possible! Diverse training opportunities and social benefits (e.g. UK pension schema) Strong hands-on experience working with modern Big Data technologies such as Apache Spark, Trino, Apache Kafka, Apache Hadoop, Apache HBase, Apache Nifi, Apache Airflow, Opensearch Proficiency in cloud-native technologies such ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

code. Perfect if you want scope for growth without going “post-technical.” What you’ll do Design and build modern data platforms using Databricks, Apache Spark, Snowflake, and cloud-native services (AWS, Azure, or GCP). Develop robust pipelines for real-time and batch data ingestion from diverse … Lead Data Engineer in complex enterprise environments. Can demonstrate strong coding skills in Python (Scala or functional languages a plus). Expertise with Databricks, Apache Spark, and Snowflake is a must (HDFS/HBase also useful). Has experience integrating large, messy datasets into reliable, scalable data products. ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

Data Engineer, working with large-scale, complex datasets in regulated environments Scala development experience supporting analytics or data-processing solutions Hands-on experience with Apache Spark for large-scale distributed data processing Experience with model calibration, threshold management, and score explainability to meet AML governance and regulatory standards … Analyst or Data Engineer, working with large, complex datasets in regulated environments Solid Scala development experience, writing and supporting scalable data and analytics solutions Apache Spark experience, including batch and/or dynamic processing of high-volume data About Us FDM is an award-winning global leader ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

Python for data engineering tasks. * Experience with cloud platforms such as AWS, Azure, or GCP. * Hands-on experience with data technologies including Apache Spark, Databricks, Kafka, Airflow, Snowflake, and Hadoop ecosystems. * Experience designing and maintaining data warehouses and data lakes. * Knowledge of DevOps and CI/CD tooling ...

subscription platforms, ad tech, and social media. Proven ability to automate and optimise data workflows , using modern ETL/ELT tools (e.g., Airflow, dbt, Apache Spark) to ensure timely and reliable delivery of data. Experience building robust data models and reporting layers to support performance dashboards, user engagement … hands-on experience in a data engineering role. Tools & Technologies : Databases : Proficient in SQL databases. Workflow Management Tools : Experience with orchestration platforms such as Apache Airflow . Programming Languages : Skilled in one or more of the following languages, i.e.: Python , Java , Go. Cloud Infrastructure : Strong understanding of cloud infrastructure ...

data platform and engineering capability as the business continues to scale its trading operations. They are looking for a Data Engineer with strong Apache Airflow experience to help design and build the pipelines that power trading, analytics and commercial decision-making. This is a great opportunity to join … platform and the opportunity to work with modern open-source technologies . What You’ll Be Doing Designing and building data pipelines orchestrated with Apache Airflow Developing ETL/ELT workflows in Python and SQL to process trading and operational data Building reliable, scalable data infrastructure to support analytics ...

architectures for ML frameworks in complex problem spaces in collaboration with product teams Experience with large scale, distributed data processing frameworks/tools like Apache Beam, Apache Spark, and cloud platforms like GCP or AWS Where You'll Be We offer you the flexibility to work where ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

understand where traditional patterns remain appropriate - and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark). Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness ...

Collaborate with senior developers and quantitative teams on new analytics capabilities Participate in debugging, testing, and production support activities Technology Environment Python Databricks/Apache Spark SQL Pandas/NumPy Git REST APIs Cloud technologies (Azure or AWS) Numerix (or equivalent pricing/risk platform) Jupyter/notebook … commercial Python development experience Strong understanding of data structures and software engineering fundamentals Experience working with data engineering or analytics tooling Exposure to Databricks, Spark, or cloud-based data platforms Understanding of SQL and relational databases Excellent problem-solving and communication skills STEM or Computer Science degree preferred Nice ...

clouds (AWS, Azure, GCP) and cloud data warehousing solutions like Snowflake and Databricks. Certifications with each are a plus. Understanding of SQL, Python, R, Apache Spark or similar product or language are also a plus.* Strong knowledge of data analytics, data science, ETL processes, AI/ML workflows ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

capacity planning, skills development, and vendor contributions The individual: Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark. Experience with data modeling, metadata and lineage tooling; performance & cost optimization. Proven leadership in engineering teams; Agile/DevOps ways of working ...

Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data Modelling You're familiar with DevOps principles, containerisation and CI/CD tools such ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

multiples sources (APIs, BDD, streams, fichiers). Qualité, Performance & Fiabilité Déployer des mécanismes de Data Quality (contrôles, monitoring, alerting). Optimiser les traitements Spark (partitioning, tuning, formats de données). Assurer la supervision et la résolution des incidents en production. CI/CD & Gouvernance Industrialiser les développements ...

integrations, while ensuring solutions meet regulatory standards and align with architectural best practices. Key Responsibilities: Build and optimise scalable data pipelines using Databricks and Apache Spark (PySpark). Ensure performance, scalability, and compliance (GxP and other standards). Collaborate on requirements, design, and backlog refinement. Promote engineering best … data pipelines that improve time-to-insight. Knowledge of secure, auditable, and compliant data workflows. Know how on optimising performance and reducing costs through Spark and Databricks tuning. Be able to create reusable, well-documented tools enabling collaboration across teams. A culture of engineering excellence driven by mentoring ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms (e.g., Snowflake, Redshift, BigQuery, Spark). Engineering Practices & Tooling Experience with CI/CD pipelines and automated build and deployment processes. Strong experience with version control systems (Git). ...

experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active SC Clearance ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...