9 of 9 Contract Delta Lake Jobs in London

building high-quality data pipelines and optimizing large-scale workloads on the Databricks Data Intelligence Platform. The ideal candidate is strong in PySpark, Delta Lake, Azure services, workflow automation, and performance tuning. Key Responsibilities Build and maintain scalable batch and streaming pipelines using Databricks notebooks, Jobs, and Workflows. … Optimize Spark and Delta Lake performance through cluster tuning, efficient configurations, AQE, and caching. Run performance tests and ensure workloads are cost-efficient and reliable. Apply data quality, lineage, and access controls using Unity Catalog and governance best practices. Develop reusable PySpark ETL and data transformation code. Manage ...

Azure & PySpark - SC Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing. Youll design and build scalable data pipelines and maintain high-quality, test-driven code … Python/PySpark pipelines for data ingestion, processing, and validation. Write unit and BDD tests using Behave, including mocking and patching. Create and optimize Delta Lake tables for reliable, performant data storage. Use Docker to manage consistent development, testing, and deployment environments. Build configurable, parameter-driven code ...

work involves developing and maintaining complex, test-driven data flows. You will be responsible for building scalable data pipelines using Python and PySpark, optimizing Delta Lake storage, and deploying solutions in a containerised Azure environment. Quality, maintainability, and automated testing are key to this role. What youll … flows with a strong focus on clean, testable Python code Writing unit and BDD tests using Behave, including mocking and patching Creating and optimizing Delta Lake tables for performance and reliability Working with Docker to support consistent development and deployment Integrating Azure services such as Azure Functions ...

workflows with strong data quality and reliability Build reusable data products in collaboration with analysts and AI engineers Manage data lakes/warehouses using Delta Lake, Parquet and related technologies Apply best practice in governance, performance and security Contribute to platform evolution, tooling adoption and engineering standards Provide … Server, Databricks Enhancements: Python (PySpark, Pandas), Azure DevOps CI/CD, Terraform/Bicep, REST APIs, GitHub/GitHub Actions Desirable: Microsoft Fabric, Delta Live Tables, Power BI automation, DataOps What We're Looking For Proven experience in data engineering or cloud data development Strong background building Azure-based ...

development, stored procedures, triggers, and query optimization; SSIS (package design, orchestration, deployment); Azure Data Factory (pipelines, triggers, integration runtimes); Azure Data Lake Storage (Gen2, security, folder structures); Microsoft Fabric (Lakehouse, OneLake, Data Engineering); Databricks (Spark SQL, notebooks, Delta Lake, integration with ADF/Fabric); - Strong data design ...

Objectives Lead data architecture and platform design on Azure Databricks + Unity Catalog, ensuring scalability, governance, and performance. Develop and optimize data pipelines (dbt, Delta Live Tables, Spark) for actuarial, claims, reference, and other data. Design and implement data models for claims (paid, incurred, case reserves, exposures), enabling actuarial … transformation roadmap. Required Skills/Experience The ideal candidate will have the following: Strong background in data architecture & engineering on Azure Databricks (Spark, Delta Lake, Unity Catalog). Hands-on experience with dbt, Delta Live Tables, Prefect, Azure DevOps CI/CD. Knowledge of actuarial data structures ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines . Start Date: ASAP Contract Type: Outside IR35 Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines . Start Date: ASAP Contract Type: Outside IR35 Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner ...

Azure & PySpark - SC Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing click apply for full job details ...