1 to 25 of 37 Delta Lake Jobs in the UK

analytics, and advanced use cases. Your Responsibilities (Up to 10, avoid repetition) Data Product & Pipeline Engineering Design, build, and maintain pipelines in Databricks using Delta Lake/Delta Live Tables Implement medallion architectures (Bronze/Silver/Gold) and deliver reusable, discoverable data products Ensure pipelines meet … skills Solid understanding of data modelling, performance tuning, and cost optimisation Desirable Skills/Knowledge/Experience Strong hands-on Databricks Lakehouse experience, including Delta Lake and Delta Live Tables (DLT) Lakehouse monitoring, data quality, and observability: DLT expectations and pipeline health monitoring Job/workflow metrics ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage. Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage . Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

models. Key Responsibilities Design, develop, and maintain end-to-end data pipelines using Microsoft Fabric and Databricks Build and optimize Lakehouse architectures using Delta Lake principles Ingest, transform, and curate data from multiple sources (APIs, databases, files, streaming) Develop scalable data transformations using PySpark and Spark SQL Implement … with Databricks in production environments Proficiency in PySpark and SQL Solid understanding of data engineering concepts (ETL/ELT, orchestration, partitioning) Experience working with Delta Lake Familiarity with cloud platforms (Azure preferred) Experience integrating data from relational and non-relational sources Knowledge of data modeling for analytics Experience ...

high-performance data solutions using Databricks and modern cloud technologies. Key Responsibilities Design, develop, and maintain scalable data pipelines using Databricks, Apache Spark, and Delta Lake Build and optimize ETL/ELT workflows for batch and streaming data Develop data models to support analytics, reporting, and machine learning … years of experience in data engineering or a similar role Strong experience with Databricks and Apache Spark Proficiency in Python and SQL Experience with Delta Lake, data modeling, and performance tuning Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with data orchestration tools (e.g., Airflow ...

data becomes a true competitive advantage. Why consider this role? Lead delivery of a strategic cloud data platform Own architecture across Azure, Databricks, Synapse & Delta Lake Influence enterprise-wide data governance and standards Hybrid working - 3 days per week in the office Competitive salary, bonus & strong benefits High … regulated finance environment What we're looking for Proven experience as a Lead/Principal Data Engineer Deep Azure data stack knowledge (Databricks, Synapse, Delta Lake, Unity Catalog) Strong SQL and data-modelling capability Experience with orchestration, monitoring and data quality tooling Background working in regulated financial services ...

modern cloud data platform. Key Responsibilities Design and implement scalable data models within Databricks. Transform raw datasets into curated, analytics-ready data products using Delta Lake and Spark. Build and maintain semantic layers for business reporting. Translate stakeholder requirements into technical specifications. Define KPIs and controlled vocabulary across … governance standards. Support CI/CD processes and version control workflows. Required Experience Strong SQL expertise (data modelling & optimisation). Hands-on Databricks experience (Delta Lake + Spark). Experience with dbt or similar transformation frameworks. Python for data manipulation and automation. Azure cloud experience. Familiarity with ...

batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance … Azure cloud services: o AWS: S3, EMR, Glue, Redshift, Lambda, IAM o Azure: ADLS Gen2, Azure Databricks, Synapse, Data Factory, Key Vault • Experience with Delta Lake, Parquet, and data modeling ...

data models, transformation logic, and quality frameworks across the data lifecycle. * Write clean, maintainable Python code for data engineering and automation tasks. * Work with Delta Lake and modern lakehouse patterns to optimise data storage and performance. * Contribute to CI/CD and DataOps practices to improve deployment reliability … years working in data engineering, data consulting, or a similar technical role. Nice to Have Skills * Exposure to Microsoft Fabric. * Experience with Delta Lake and lakehouse architectures. * Familiarity with CI/CD, DataOps, or modern deployment pipelines. Soft Skills * Confident in client facing interactions, including workshops, requirements gathering ...

forward-thinking, data-driven organisation. If you thrive bridging the gap between data engineering and analytics, and enjoy working with tools like Databricks, Delta Lake, Spark, and dbt, this is an exciting short-term opportunity. What You'll Do: Transform raw data into structured, reliable datasets … workflows, SQL queries, and Databricks pipelines for performance. What We're Looking For: Strong SQL and data modelling skills. Hands-on experience with Databricks, Delta Lake, Spark, and preferably dbt. Python for data manipulation and automation. Familiarity with BI tools (Power BI) and cloud platforms (Azure). Understanding ...

Data Engineer/Data Engineering/Data Consultant/Lakehouse/Delta Lake/Data Warehousing/ETL/Azure/Azure Databricks/Python/SQL/Based in the West Midlands/Solihull/Birmingham area, Permanent role, £ + car/allowance (£5,000) + … allowance + bonus Experience: Experience in a Data Engineer/Data Engineering role Large and complex datasets Azure, Azure Databricks Microsoft SQL Server Lakehouse, Delta Lake Data Warehousing ETL Database Design Python/PySpark Azure Blob Storage Azure Data Factory Desirable: Exposure ML/Machine Learning/ ...

quality, observability, and cost-efficient performance. Responsibilities • Design and implement complex ETL/ELT pipelines using Databricks (Python, SQL, DLT). • Build and optimise Delta Lake tables with effective partitioning and performance strategies. • Contribute to Lakehouse architecture design, ingestion patterns, and data product boundaries. • Ensure solutions align with … What You'll Need to Be Successful • Strong experience as a Senior Data Engineer in modern data environments. • Deep expertise in Databricks, Spark, Delta Lake, and Lakehouse architectures. • Proven ability to design scalable, modular, and maintainable data solutions. • Experience working in cloud platforms (Azure preferred). • Solid understanding ...

quality, observability, and cost-efficient performance. Responsibilities Design and implement complex ETL/ELT pipelines using Databricks (Python, SQL, DLT). Build and optimise Delta Lake tables with effective partitioning and performance strategies. Contribute to Lakehouse architecture design, ingestion patterns, and data product boundaries. Ensure solutions align with … What You'll Need to Be Successful Strong experience as a Senior Data Engineer in modern data environments. Deep expertise in Databricks, Spark, Delta Lake, and Lakehouse architectures. Proven ability to design scalable, modular, and maintainable data solutions. Experience working in cloud platforms (Azure preferred). Solid understanding ...

tests (Behave), including mocking and patching Using PySpark to process large-scale datasets while keeping business logic in Python Creating, maintaining, and optimising Delta Lake tables for performance and reliability Building and running applications in containerised (Docker) environments Integrating Python applications with Azure services such as Azure Functions … Python development experience (essential) Proven experience writing test-driven Python code in production environments Solid data engineering experience using PySpark Experience working with Delta Lake Hands-on experience with Docker and containerised workflows Good knowledge of Azure and integrating Python applications with cloud services A software-engineering mindset ...

with architectural ownership. Key responsibilities Design and architect a secure, scalable AWS-based data platform using Databricks Build and operate pipelines using Databricks, PySpark, Delta Lake, and CI/CD Assess the existing data landscape and define migration approaches Lead early data migrations from legacy platforms into … communication and documentation are essential. What good looks like Significant hands-on experience with Databricks (engineering and architecture) Strong practical skills in: Databricks PySpark Delta Lake Data modelling and performance optimisation CI/CD pipelines Comfortable operating in a complex, programme-driven environment Able to clearly explain technical ...

ability to translate business requirements into technical solutions. Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: Airflow, DBT Data Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Infrastructure and Operations: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable Skills and Experience Experience with enterprise data platforms ...

with excellent communication skills. Technical Stack: Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: MLflow, Azure ML, Airflow, Docker, Fargate, FastAPI Data: Snowflake, Delta Lake, Redis, Azure Data Lake Infra & Ops: Terraform, GitHub Actions, Azure DevOps, Grafana, Azure Monitor Desirable skills/knowledge/experience: Knowledge … data platforms (e.g. Snowflake, Azure Data Lake). Experience deploying models as APIs (FastAPI, Azure Functions). Understanding of monitoring, model performance tracking, and observability best practices. ...

Experience working with or supporting cloud migrations. Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: Airflow, DBT, Docker, Fargate, FastAPI Data Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Infrastructure and Operations: Terraform, GitHub Actions, Azure DevOps, Grafana, Azure Monitor Desirable Skills and Experience Experience working ...

help modernise legacy data estates and embed strong data engineering standards. Key Responsibilities: Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance ...

data engineering, data platforms and big data technologies Evaluate and improved existing data processes Essential Experiences: Advanced SQL skills Proficient in Databricks, including Spark, Delta Lake and MLflow Experience migrating SSIS to Databricks In depth understanding of ETL/ELT processes Knowledge of data warehousing concepts and architectures ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

improvement across the data engineering function. Responsibilities Build and maintain scalable ETL/ELT pipelines using Databricks (SQL, Python, Spark). Develop and optimise Delta Lake tables across medallion layers. Implement best practices for performance, modularity, and reusability. Support CI/CD processes for notebooks, jobs, and data ...

improvement across the data engineering function. Responsibilities Build and maintain scalable ETL/ELT pipelines using Databricks (SQL, Python, Spark). Develop and optimise Delta Lake tables across medallion layers. Implement best practices for performance, modularity, and reusability. Support CI/CD processes for notebooks, jobs, and data ...

across the organisation. You'll design and develop scalable data pipelines within a modern Azure ecosystem, working with technologies such as Azure Synapse, Databricks, Delta Lake, Unity Catalog, and Databricks SQL. Responsibilities include overseeing pipeline orchestration, implementing data quality controls, monitoring and alerting, and managing data lineage, glossaries ...