1 to 25 of 42 Delta Lake Jobs in the UK

frameworks and modern data engineering solutions Provide technical leadership and architectural guidance across data platform initiatives Design and govern Lakehouse architectures leveraging Databricks and Delta Lake Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams Identify and mitigate delivery risks, dependencies and technical challenges Troubleshoot complex … Technical Lead, Engineering Lead, or similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture ...

/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure data security, governance, and compliance (access control, encryption, lifecycle) Provide technical leadership, mentoring and design reviews Collaborate with stakeholders … analysis, profiling and insight delivery Cloud & Platforms Azure: Data Factory, Synapse, Databricks, ADLS Gen2, Event Hub, Purview AWS: S3, Glue, Athena, Kinesis, Lambda, EMR, Lake Formation Databricks: Delta Lake, Unity Catalog, Structured Streaming, MLflow SQL Server stack (SSIS, SSRS) Engineering Practices CI/CD for data platforms ...

Build and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF). Develop Change Data Capture (CDC) solutions to support incremental and delta data loads. Create data transformation and cleansing processes using PySpark and Spark DataFrames. Develop Python-based utilities and automation scripts to support migration … optimise data models, SQL queries, stored procedures and analytical views within Azure Synapse Analytics. Manage data ingestion, transformation and storage across Azure Data Lake Storage (ADLS) and Azure SQL environments. Support processing of structured and semi-structured data formats, including Delta Lake and JSON. Collaborate with stakeholders ...

global environment. Description Design, build and maintain scalable data pipelines using Python and Databricks Develop and optimise ETL/ELT processes leveraging Spark and Delta Lake Create robust data models and data architectures within modern Lakehouse environments Implement and manage data workflows across the Databricks ecosystem Ensure data … Capital Markets, Insurance, or other highly regulated enterprise environments. Strong hands-on expertise in Python Proven experience with Databricks, including notebooks, workflows, jobs and Delta Lake Strong knowledge of Apache Spark/PySpark Experience designing and delivering large-scale ETL/ELT pipelines Advanced SQL skills and experience ...

dimensional modelling) for consistent implementation across data solutionsEvaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patternsDesign technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)Create and maintain architecture documentation, including data flow diagrams, architecture … recommend new technologies and tools that enhance Arriva's data and BI capabilitiesStay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutionsConduct proofs-of-concept and technical assessments to validate new technologies before adoptionMonitor ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

Data Engineer will design, build, support and operate data pipelines and core data platform technologies within the organisation's Azure-based analytics and lake house environment. The role is hands-on and implementation-focused, working within established architectural patterns to deliver reliable, secure and well-governed data for analytics … Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including performance monitoring and configuration Build Gold-layer datasets optimised for Power ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

environment strategy. Design and govern data integration patterns for sharing data with external clients and third parties, including secure, scalable approaches using Databricks Delta Sharing, APIs, and file-based exchange where appropriate. Lead the design and adoption of Master Data Management (MDM), including entity definitions, golden-record strategy, match … merge rules, and stewardship processes. Establish standards and practices for optimizing Databricks performance and cost — cluster and compute strategy, Delta table management, caching, and workload right-sizing. Define data modelling standards (conceptual, logical, physical) across analytical and operational domains. Collaborate with data governance to embed data quality, lineage, cataloguing ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

solutions end-to-end, with a focus on Lakehouse Architecture and ML/AI-driven analytics. Design scalable data models and pipelines using Spark, Delta Lake, and PySpark. Lead client engagements, shaping data strategies and best‐practice architectures. Win business through presenting solutions and advising potential clients. Optimise ...

data fidelity. Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

model lifecycle: data prep, experimentation, deployment, and monitoring.* Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS.* Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing.* Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering.* Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering.* Experience with Databricks, Spark, and Delta Lake.* Strong SQL skills with experience working on large datasets.* Experience with AWS, including S3 and Lambda.* Familiarity with Redshift, Snowflake, or other cloud ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. ...

into technical solutions Requirements: Enterprise data warehouse architecture and design Microsoft Fabric, Azure Data Platform technologies and modern Lakehouse architectures Databricks, Unity Catalog and Delta Lake PySpark, Databricks, SQL Medallion Architecture implementation Data modelling approaches including Star Schema and Data Vault Data governance, security and access management ...

Views, SLT, or equivalent SAP-native frameworks. Design and implement Medallion-aligned data pipelines (Bronze/Silver/Gold) in our cloud data lake or lakehouse (Azure/AWS/GCP/Databricks/Snowflake). Assess and remediate the current third-party implementation , identifying blockers and accelerating delivery. … strategy , including prioritisation of modules (FI/CO/MM/SD/PP etc.) required for December cutover. Build scalable ingestion pipelines using Delta Lake/Parquet and modern orchestration tools (ADF, dbx, Airflow, etc.). Collaborate with SAP functional teams to validate business logic, deltas ...

creation of data domains that align with the banks underlying capabilities Data Engineering & Platforms Design data product schemas across Snowflake and Databricks, including Delta Lake structures and access patterns Define and enforce architectural guardrails for data product design, including performance, scalability, security, and cost efficiency Set design standards ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn, and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...