1 to 25 of 63 Permanent Apache Spark Jobs in London

Python ML Model Development Kafka,Spark, PostgreSQL scikit-learn, PyTorch, pandas, NumPy, XGBoost, Spark, Apache Spark,Apache Airflow,Apache Kafka, RabbitMQ,PostgreSQL, MySQL R, Bash, TensorFlow, XGBoost, LightGBM, Mllib,Classical ML, Deep Learning, NLP, Time Series Forecasting, DataBricks ...

possible! Diverse training opportunities and social benefits (e.g. UK pension schema) Strong hands-on experience working with modern Big Data technologies such as Apache Spark, Trino, Apache Kafka, Apache Hadoop, Apache HBase, Apache Nifi, Apache Airflow, Opensearch Proficiency in cloud-native technologies such ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … code generation in regulated environments Strong hands-on experience with AWS ecosystem: AWS Glue, S3, and related data platforms (e.g. Snowflake) Strong experience with Apache Airflow and DAG-based orchestration Experience designing configuration-driven frameworks: Using YAML or JSON Experience building automated testing for data pipelines: Functional equivalence testing ...

data engineering tasks. Experience with cloud platforms such as: AWS, Azure, Google Cloud Platform (GCP) Hands-on experience with data technologies such as: Apache Spark, Databricks, Kafka, Airflow, Snowflake, Hadoop ecosystems Experience designing and maintaining data warehouses and data lakes. Knowledge of DevOps and CI/CD tooling ...

subscription platforms, ad tech, and social media. Proven ability to automate and optimise data workflows , using modern ETL/ELT tools (e.g., Airflow, dbt, Apache Spark) to ensure timely and reliable delivery of data. Experience building robust data models and reporting layers to support performance dashboards, user engagement … hands-on experience in a data engineering role. Tools & Technologies : Databases : Proficient in SQL databases. Workflow Management Tools : Experience with orchestration platforms such as Apache Airflow . Programming Languages : Skilled in one or more of the following languages, i.e.: Python , Java , Go. Cloud Infrastructure : Strong understanding of cloud infrastructure ...

architectures for ML frameworks in complex problem spaces in collaboration with product teams Experience with large scale, distributed data processing frameworks/tools like Apache Beam, Apache Spark, and cloud platforms like GCP or AWS Where You'll Be We offer you the flexibility to work where ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

years (Senior) of data engineering experience Hands-on experience with Palantir Foundry and/or Foundry Data Platform (FDP) Strong proficiency in: Python SQL Spark (PySpark or SparkSQL) Experience building production data pipelines in cloud environments (AWS, Azure, or GCP) Understanding of data modeling, schema design, and metadata management ...

disparate sources to build meaningful datasets for analytics and reporting, focusing on consolidating data from various systems. Implement big-data technologies (e.g., Redshift, EMR, Spark, SNS, SQS, Kinesis) to optimize processing of large datasets. Develop and maintain the team's data platform, including infrastructure-as-code using AWS CDK. … permissions Experience with non-relational databases/data stores (object storage, document or key-value stores, graph databases, column-family databases) Experience with Apache Spark/Elastic Map Reduce ...

clouds (AWS, Azure, GCP) and cloud data warehousing solutions like Snowflake and Databricks. Certifications with each are a plus. Understanding of SQL, Python, R, Apache Spark or similar product or language are also a plus.* Strong knowledge of data analytics, data science, ETL processes, AI/ML workflows ...

with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark or other big-data tools Hybrid working – 3 days per week in a central London office (Tues - Thurs) If you’re excited ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

multiples sources (APIs, BDD, streams, fichiers). Qualité, Performance & Fiabilité Déployer des mécanismes de Data Quality (contrôles, monitoring, alerting). Optimiser les traitements Spark (partitioning, tuning, formats de données). Assurer la supervision et la résolution des incidents en production. CI/CD & Gouvernance Industrialiser les développements ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design ...

Cloud engineering: Azure (preferred) and/or AWS/GCP; Kubernetes/Docker; serverless (Azure Functions/Lambda); secure networking/IAM. Data engineering: Spark/Databricks, ETL/ELT pipelines; integration with client data sources and enterprise systems; runbooks and operational handover. Product delivery: workshop facilitation, PRDs, user ...

improve data engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance … data pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ...

looking for Strong Python + SQL (essential) 5+ years in data engineering/platform engineering AWS experience in production data environments Hands-on with Spark/Kafka/Snowflake/Databricks (or similar) Experience with data lakes/lakehouse/distributed systems Background in financial services, trading, or quantitative ...

scalable data pipelines, managing large datasets, and enabling engineering and AI teams to deliver high-performance data-driven products. Key Technical Requirements • Python • SQL • Spark • Airflow • Data pipelines • AWS or GCP Experience Required • 4+ years in data engineering • Experience working with large-scale structured and unstructured datasets • Strong ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

experience productionising machine learning models Are an expert in at least one of: predictive modelling, classification, regression, optimisation or recommendation systems Have experience with Spark Have knowledge of DevOps technologies such as Docker and Terraform and ML Ops practices and platforms like ML Flow Have experience with agile delivery ...

through influence with strong collaborator engagement skills. Preferred Experience Digital Twin, asset virtualization, or industrial data platforms Graph technologies (e.g., Neo4j), big data ecosystems (Spark, Hive, Hadoop) Databricks, Python, and modern analytics platforms Delivery of data products at enterprise scale Key Competencies Strategic product thinking with strong execution bias ...

Skills (Bonus Points) Experience with MLOps tooling (MLflow, Kubeflow, Weights & Biases, or similar). Building data pipelines (real‐time or batch) using tools like Apache Spark, Kafka, Airflow, or dbt. Familiarity with recommender systems, transformers, or graph neural networks. Exposure to model optimisation techniques (quantisation, distillation, efficient inference ...

internal knowledge Competencies and Behaviours: 3+ years experience in data platform engineering, cloud engineering, or similar roles Strong hands-on experience with Databricks, including Apache Spark, Delta Lake, Workflows Proven experience designing and deploying data platforms on AWS and/or Azure (e.g. ADLS, S3, VNet/ ...

bring: Strong SQL + ETL/ELT experience Knowledge of Python/T-SQL Exposure to cloud platforms (Azure preferred) Nice to have: Databricks, Spark, CI/CD, big data This is a great opportunity to join a high-impact data team in a global organisation and shape ...