1 to 25 of 38 Delta Lake Jobs in the UK

reliable delivery of new data pipelines alongside ongoing operational stability.This is a hands-on engineering role focused on implementation, optimisation and support of Delta Lake pipelines across Bronze, Silver, Staging and Gold layers, operating within an established architectural framework. Key Responsibilities Develop and maintain data pipelines in Azure … Databricks using PySpark and SQL Build incremental processing using Delta Lake and Change Data Feed Extend and maintain watermark-driven CDC frameworks Implement and maintain Slowly Changing Dimension (SCD Type 2) logic Engineer time-aware joins and dimensional transformations Support optimisation of Spark workloads and Delta tables ...

working on behalf of a major client in the energy sector. The position focuses on designing, developing, and maintaining Azure-based data platforms-particularly Delta Lake and modern analytics services-while providing architectural leadership and hands-on engineering expertise. The role operates on a hybrid model, requiring three … days per week onsite in Glasgow. Key Responsibilities Develop and maintain Azure data architecture , including Delta Lake, ensuring compliance with data governance and security standards. Analyse and resolve data integration and system interoperability challenges across complex environments. Design and implement Azure data solutions using services such as Databricks ...

data into structured, analytics-ready formats. Optimise pipeline performance and ensure high availability of data services. Infrastructure & Architecture Architect and deploy scalable data lake solutions using Azure Data Lake Storage. Implement governance and security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible … deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations and CI/CD pipelines for Databricks deployments. Monitoring & Operations Implement monitoring solutions using Azure ...

work at home 100% of the time. Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance … continuous improvement. Your Skills: Expert-level hands-on experience with Azure Data Services and Databricks in enterprise, highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing ...

data becomes a true competitive advantage. Why consider this role? Lead delivery of a strategic cloud data platform Own architecture across Azure, Databricks, Synapse & Delta Lake Influence enterprise-wide data governance and standards Hybrid working - 3 days per week in the office Competitive salary, bonus & strong benefits High … regulated finance environment What we're looking for Proven experience as a Lead/Principal Data Engineer Deep Azure data stack knowledge (Databricks, Synapse, Delta Lake, Unity Catalog) Strong SQL and data-modelling capability Experience with orchestration, monitoring and data quality tooling Background working in regulated financial services ...

modern cloud data platform. Key Responsibilities Design and implement scalable data models within Databricks. Transform raw datasets into curated, analytics-ready data products using Delta Lake and Spark. Build and maintain semantic layers for business reporting. Translate stakeholder requirements into technical specifications. Define KPIs and controlled vocabulary across … governance standards. Support CI/CD processes and version control workflows. Required Experience Strong SQL expertise (data modelling & optimisation). Hands-on Databricks experience (Delta Lake + Spark). Experience with dbt or similar transformation frameworks. Python for data manipulation and automation. Azure cloud experience. Familiarity with ...

data models, transformation logic, and quality frameworks across the data lifecycle. * Write clean, maintainable Python code for data engineering and automation tasks. * Work with Delta Lake and modern lakehouse patterns to optimise data storage and performance. * Contribute to CI/CD and DataOps practices to improve deployment reliability … years working in data engineering, data consulting, or a similar technical role. Nice to Have Skills * Exposure to Microsoft Fabric. * Experience with Delta Lake and lakehouse architectures. * Familiarity with CI/CD, DataOps, or modern deployment pipelines. Soft Skills * Confident in client facing interactions, including workshops, requirements gathering ...

forward-thinking, data-driven organisation. If you thrive bridging the gap between data engineering and analytics, and enjoy working with tools like Databricks, Delta Lake, Spark, and dbt, this is an exciting short-term opportunity. What You'll Do: Transform raw data into structured, reliable datasets … workflows, SQL queries, and Databricks pipelines for performance. What We're Looking For: Strong SQL and data modelling skills. Hands-on experience with Databricks, Delta Lake, Spark, and preferably dbt. Python for data manipulation and automation. Familiarity with BI tools (Power BI) and cloud platforms (Azure). Understanding ...

with architectural ownership. Key responsibilities Design and architect a secure, scalable AWS-based data platform using Databricks Build and operate pipelines using Databricks, PySpark, Delta Lake, and CI/CD Assess the existing data landscape and define migration approaches Lead early data migrations from legacy platforms into … communication and documentation are essential. What good looks like Significant hands-on experience with Databricks (engineering and architecture) Strong practical skills in: Databricks PySpark Delta Lake Data modelling and performance optimisation CI/CD pipelines Comfortable operating in a complex, programme-driven environment Able to clearly explain technical ...

shape the Society’s long-term data strategy. About you - Extensive Technical Expertise : Strong knowledge of Microsoft Fabric components including OneLake , Lakehouse/Warehouse, Delta Lake, Direct Lake, Data Factory, Spark ( PySpark/Scala ) and Power BI (DAX and semantic modelling). Advanced Programming and Data Engineering ...

PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure acceptable) Understanding of data lake and lakehouse architectures Ability to work autonomously and collaboratively within delivery squads Experience working within Central Government departments (highly desirable) Desirable Azure Data Lake, Synapse, Delta Lake Streaming technologies (Kafka, Event Hubs ...

ability to translate business requirements into technical solutions. Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: Airflow, DBT Data Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Infrastructure and Operations: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable Skills and Experience Experience with enterprise data platforms ...

work cross-functionally with stakeholders, engineers, and analysts. Technical Stack: Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: Airflow, DBT, Docker Data: Snowflake, Delta Lake, Redis, Azure Data Lake Infra & Ops: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable skills/knowledge/experience: Knowledge … data platforms (e.g. Snowflake, Azure Data Lake). Experience deploying models as APIs (FastAPI, Azure Functions). Understanding of monitoring, model performance tracking, and observability best practices. Familiarity with orchestration tools like Airflow or Azure Data Factory. ...

Strong analytical mindset with the ability to drive business impact through data Technical Environment Languages: Python, SQL, Bash Cloud: Azure, AWS Data & Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Tools: Airflow, DBT, Docker Infrastructure & Operations: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable Skills Experience with … data platforms such as Snowflake or Azure Data Lake Experience deploying models as APIs (e.g. FastAPI, Azure Functions) Knowledge of monitoring, model performance tracking and observability practices Familiarity with orchestration tools such as Airflow or Azure Data Factory ...

with excellent communication skills. Technical Stack: Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: MLflow, Azure ML, Airflow, Docker, Fargate, FastAPI Data: Snowflake, Delta Lake, Redis, Azure Data Lake Infra & Ops: Terraform, GitHub Actions, Azure DevOps, Grafana, Azure Monitor Desirable skills/knowledge/experience: Knowledge … data platforms (e.g. Snowflake, Azure Data Lake). Experience deploying models as APIs (FastAPI, Azure Functions). Understanding of monitoring, model performance tracking, and observability best practices. ...

Experience working with or supporting cloud migrations. Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: Airflow, DBT, Docker, Fargate, FastAPI Data Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Infrastructure and Operations: Terraform, GitHub Actions, Azure DevOps, Grafana, Azure Monitor Desirable Skills and Experience Experience working ...

help modernise legacy data estates and embed strong data engineering standards. Key Responsibilities: Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance ...

engineering with Apache Airflow, dbt, and Prefect, ensuring performance, governance, and alignment with enterprise metadata standards (Collibra, Alation, Microsoft Purview). Drive adoption of Delta Lake, Iceberg, and Hudi to support scalable data lakehouse architectures. Ensure high‐quality, compliant, and reliable data foundations for ML and analytics workloads. ...

business. Contribute to architectural discussions and the ongoing data platform roadmap. Tech You'll Use Databricks: Lakeflow Declarative Pipelines, Lakeflow Workflows, Unity Catalog, Delta Lake Azure: ADLS Gen2, Data Factory, Event Hubs (optional), Key Vault, private endpoints Languages: PySpark, Spark SQL, Python, Git DevOps: Azure DevOps Repos & Pipelines … Azure + Databricks . Strong PySpark and Spark SQL skills within distributed compute environments. Experience working in a Lakehouse/Medallion architecture with Delta Lake. Understanding of dimensional modelling (Kimball), including SCD Type 1/2. Exposure to operational concepts such as monitoring, retries, idempotency and backfills. Mindset Good ...

line with the organisation’s norms and values Essential • Proven experience in managing data operations, integrations, and engineering teams. • Strong expertise in Databricks, including Delta Lake, Spark, and automated pipeline development. • Hands-on experience with ETL/ELT processes, data governance, and cloud-based data platforms. • Proficiency ...

Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake architecture, delta lake tables, security policies, and data governance within Fabric. o Build scalable, reusable data assets and engineering patterns that support analytics, reporting ...

cloud data strategy, define engineering best practice and guide delivery across cross-functional teams. Responsibilities Build and optimise Azure data platforms using ADF, Data Lake, Azure Functions and Databricks Deliver reliable, high-quality data pipelines Work with stakeholders to ensure solutions meet Lloyd's of London regulatory, audit … Data Services and Databricks in enterprise environments. Proven delivery experience within the Lloyd's of London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record ...

across the organisation. You'll design and develop scalable data pipelines within a modern Azure ecosystem, working with technologies such as Azure Synapse, Databricks, Delta Lake, Unity Catalog, and Databricks SQL. Responsibilities include overseeing pipeline orchestration, implementing data quality controls, monitoring and alerting, and managing data lineage, glossaries ...

wont just be delivering features; youll be setting direction. The tech youll work with Core:Azure, Databricks, Python, SQL, dbt, MQTT Storage & Serving:Delta Lake, Postgres, TimescaleDB Modelling & ML:MLflow Visualisation:Grafana What makes you a great fit Youre someone who can zoom out to architecture and zoom ...

consumption layers Technology Leadership Evaluate and recommend modern data technologies and tooling Stay current with industry trends such as open table formats (Apache Iceberg, Delta Lake), data observability, and cloud-native services Lead proofs-of-concept and technical assessments for new technologies Ensure the data architecture remains modern ...