17 of 17 Contract Apache Spark Jobs in London

Data Platform Engineer - ADF, Terraform, Synapse, Spark £Market Rate (Inside IR35) London/Hybrid 6 months We are currently working with a client who urgently require a Data Platform Engineer with hands on expertise in Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing and delivering … Data Platform Engineer Advanced knowledge of Azure Data Factory (ADF), including network configurations and integration runtime options. Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts. Experience in efficient handling of large-scale data Deep understanding of Azure DevOps pipelines and deploying Terraform (Infrastructure ...

data warehousing concepts, and distributed computing. Familiarity with CI/CD, version control, and DevOps practices. Nice-to-Have Experience with streaming technologies (e.g., Spark Structured Streaming, Event Hub, Kafka). Knowledge of MLflow, Unity Catalog, or advanced Databricks features. Exposure to Terraform or other IaC tools. Experience working ...

data warehousing concepts, and distributed computing. Familiarity with CI/CD, version control, and DevOps practices. Nice-to-Have Experience with streaming technologies (e.g., Spark Structured Streaming, Event Hub, Kafka). Knowledge of MLflow, Unity Catalog, or advanced Databricks features. Exposure to Terraform or other IaC tools. Experience working ...

development. Hands-on experience with Azure Data Factory, Azure Data Lake, and Azure SQL Database. Exposure to big data technologies such as Hadoop, Spark, and Databricks. Experience with Azure Synapse Analytics or Cosmos DB. Familiarity with data governance frameworks (e.g., GDPR, HIPAA). Experience implementing CI/CD pipelines ...

integration patterns. Experience of working with complex data pipelines, large data sets, data pipeline optimization + data architecture design. Implementing complex data transformations using Spark, PySpark or Scala + working with SQL/MySQL databases. Experience with data quality, data governance processes, Git version control + Agile development environments. ...

years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks . Good proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure ...

10+ years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. ·Good proficiency in Python and Spark (PySpark) or Scala. ·Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. ·Extensive experience with Azure data services, including Azure ...

automate robust CI/CD pipelines using GIT to ensure stable, reliable, and frequent model releases. Performance Engineering: Profile and optimise large-scale Spark/Python codebases for production efficiency, focusing on minimising latency and cost. Knowledge Transfer: Act as the technical lead to embed MLOps standards into … MLOps processes in a production environment. Cloud ML Stack: Expert proficiency with Databricks and MLflow . Big Data/Coding: Expert Apache Spark and Python engineering experience on large datasets. Core Engineering: Strong experience with GIT for version control and building CI/CD/release pipelines. Data ...

Optimise data workflows for performance, cost efficiency, and maintainability using modern data-engineering tools and platforms (e.g., Azure Data Factory, AWS Data Pipeline, Databricks, Apache Spark). Support the integration of data into visualisation platforms and analytical environments (e.g., Power BI, ServiceNow). Ensure adherence to data governance ...

pipelines and contribute to digital transformation initiatives across government departments. Key Responsibilities Design, develop and maintain robust data pipelines using PostgreSQL and Airflow or Apache Spark Collaborate with frontend/backend developers using Node.js or React Implement best practices in data modelling, ETL processes and performance optimisation Contribute … DevOps practices What We're Looking For Proven experience as a Data Engineer in complex environments Strong proficiency in PostgreSQL and either Airflow or Spark Solid understanding of Node.js or React for integration and tooling Familiarity with containerisation technologies (Docker/Kubernetes) is a plus Excellent communication and stakeholder ...

data extraction, transformation, and pipeline development. Experience with dashboarding and visualization tools (Tableau, Qlik, or similar). Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes. Useful experience; Python or R for advanced analytics, automation, or experimentation support. Knowledge of statistical methods and experimentation (A/ ...

profiling techniques to ensure accuracy and consistency Implement access controls, data masking, and compliance-aligned security protocols Tune workloads and optimise performance across Spark, Fabric, and Azure components Translate business requirements into technical solutions through close collaboration with analysts and stakeholders Maintain clear documentation and contribute to internal knowledge … repositories Essential Skills Strong experience developing within Microsoft Azure and Microsoft Fabric: Proficiency in Spark programming including DataFrames, RDDs, and Spark SQL Python and PySpark development experience, including notebook-based workflows Hands-on experience with Spark streaming and batch processing Delta table optimisation and Fabric Spark ...

roles or collaboration with trading desks Familiarity with financial instruments across asset classes (equities, FX, fixed income, derivatives) Experience with distributed computing frameworks (e.g., Spark, Dask) and cloud-native ML pipelines Exposure to LLMs, graph learning, or other advanced AI methods Strong publication record or open-source contributions ...

roles or collaboration with trading desks Familiarity with financial instruments across asset classes (equities, FX, fixed income, derivatives) Experience with distributed computing frameworks (e.g., Spark, Dask) and cloud-native ML pipelines Exposure to LLMs, graph learning, or other advanced AI methods Strong publication record or open-source contributions ...

financial data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks. Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle. Work extensively with Unity Catalog, Delta Lake, Spark ...

orchestrate data pipelines across Snowflake and AWS environments. Apply data modeling, warehousing, and architecture principles (Kimball/Inmon). Develop pipeline programming using Python, Spark, and SQL; integrate APIs for seamless workflows. Support Machine Learning and AI initiatives, including NLP, Computer Vision, Time Series, and LLMs. Implement MLOps … data pipeline development and orchestration. Proficient with cloud platforms (Snowflake, AWS fundamentals). Solid understanding of data architecture, warehousing, and modeling. Programming expertise: Python, Spark, SQL, API integration. Knowledge of ML/AI frameworks, MLOps, and advanced analytics concepts. Experience with CI/CD, data testing frameworks, and versioning ...

resource utilisation, and capacity planning. Troubleshoot platform issues and ensure SLAs for uptime and reliability. Manage services such as HDFS, YARN, Hive, Impala, Kafka, Spark, Ranger, Knox, ZooKeeper, etc. Handle cluster upgrades, patching, backups, security hardening, and user provisioning. Implement governance, auditing, and access control following best practices. Automate … incident resolution. Required Skills & Experience Strong experience as a Cloudera Administrator in production environments. Solid knowledge of Hadoop ecosystem services (HDFS, YARN, Hive, Impala, Spark ). Expertise with Cloudera Manager , cluster configuration, and life cycle management. Strong Linux administration skills (RedHat/CentOS/Ubuntu). Good understanding ...