5 of 5 Contract Delta Lake Jobs in London

analytics, and advanced use cases. Your Responsibilities (Up to 10, avoid repetition) Data Product & Pipeline Engineering Design, build, and maintain pipelines in Databricks using Delta Lake/Delta Live Tables Implement medallion architectures (Bronze/Silver/Gold) and deliver reusable, discoverable data products Ensure pipelines meet … skills Solid understanding of data modelling, performance tuning, and cost optimisation Desirable Skills/Knowledge/Experience Strong hands-on Databricks Lakehouse experience, including Delta Lake and Delta Live Tables (DLT) Lakehouse monitoring, data quality, and observability: DLT expectations and pipeline health monitoring Job/workflow metrics ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage. Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

tests (Behave), including mocking and patching Using PySpark to process large-scale datasets while keeping business logic in Python Creating, maintaining, and optimising Delta Lake tables for performance and reliability Building and running applications in containerised (Docker) environments Integrating Python applications with Azure services such as Azure Functions … Python development experience (essential) Proven experience writing test-driven Python code in production environments Solid data engineering experience using PySpark Experience working with Delta Lake Hands-on experience with Docker and containerised workflows Good knowledge of Azure and integrating Python applications with cloud services A software-engineering mindset ...

help modernise legacy data estates and embed strong data engineering standards. Key Responsibilities: Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance ...

freshness. Any AI/ML knowledge specifically around LLM's/RAG would also be useful. Data Engineer Key Skills: Python/Pyspark Databricks Delta Lake Azure LLM's/RAG Location: Remote Rate: Circa £450/Day (Outside) Contract Length: 6-12 Months Initial If this Data ...