1 to 25 of 158 Delta Lake Jobs in the UK

optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data platforms. Lead … deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

data engineering platform Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security … boards The candidate Experience and knowledge requirements: Required Experience in data engineering disciplines, with advanced, hands‐on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governance Extensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehouses ...

data engineering platform• Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory• Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security … architectural review boardsThe candidateExperience and knowledge requirements:RequiredExperience in data engineering disciplines, with advanced, hands-on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governanceExtensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehousesExpert-level proficiency ...

Asset Bundles. Key Responsibilities Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design and manage Delta Tables, including … workflows. Package and deploy Databricks resources across environments using Databricks Asset Bundles and GitLab-based CI/CD pipelines. Integrate Databricks with Azure Data Lake Storage and Azure Data Factory for secure, reliable data processing. Optimize Spark workloads, clusters, jobs, and storage patterns for performance, scalability, reliability, and cost ...

GitHub Actions and Terraform setup for deployments Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture Work closely with analysts … they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows Tech stack Azure Databricks (Spark, Delta Lake) SQL Server, Confluent Kafka Scala, SQL, dbt Git, GitHub Actions, Terraform Sigma (BI) Must-have experience Commercial experience as a Data Engineer ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

Data Engineer, you'll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks. Responsibilities: Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue. Data Transformation: Develop, maintain … optimize transformation pipelines within our Delta Lake on Databricks. Data Modelling: Design and implement efficient and reliable data models to store and retrieve data. Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions. Data Quality: Ensure the reliability, integrity ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

considered for future opportunities that match your expertisePrimary Responsibilities• Client Engagement & Delivery• Data Pipeline Development (Batch and Streaming)• Databricks & Lakehouse Architectures• Data Modelling & Optimisation (Delta Lake, Medallion architecture)• Collaboration & Best Practices• Quality, Governance & SecurityBusiness Relationships• Solution Architects• Data Engineers, Developers, ML Engineers, and Analysts• Client stakeholders … mentoring teams of engineers to deliver high-quality scalable data solutions.Technical Expertise• Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow).• Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent.• Strong SQL and Python (or equivalent ...

Primary Responsibilities Client Engagement & Delivery Data Pipeline Development (Batch and Streaming) Databricks & Lakehouse Architectures Data Modelling & Optimisation (Delta Lake, Medallion architecture) Collaboration & Best Practices Quality, Governance & Security Business Relationships Solution Architects Data Engineers, Developers, ML Engineers, and Analysts Client stakeholders up to Head of Data Engineering, Chief Data … mentoring teams of engineers to deliver high‐quality scalable data solutions. Technical Expertise Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow). Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent. Strong SQL and Python ...

grow, belong and thrive.Your day at NTT DATAPrimary Responsibilities• Client Engagement & Delivery• Data Pipeline Development (Batch and Streaming)• Databricks & Lakehouse Architectures• Data Modelling & Optimisation (Delta Lake, Medallion architecture)• Collaboration & Best Practices• Quality, Governance & SecurityBusiness Relationships• Solution Architects• Data Engineers, Developers, ML Engineers, and Analysts• Client stakeholders … mentoring teams of engineers to deliver high-quality scalable data solutions.Technical Expertise• Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow).• Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent.• Strong SQL and Python (or equivalent ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

practices.What You'll DoArchitect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments.Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate.Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration.What We're Looking ForProven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments.Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data platforms.Extensive experience ...

DoArchitect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments. Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate. Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration. What We're Looking ForProven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data ...

engineering challenges, and driving best practice across the platform. What you'll be doing Designing, building and optimising Databricks pipelines and jobs Creating scalable Delta Lake solutions using Medallion architecture Developing and maintaining Azure Data Factory pipelines Owning data models across bronze, silver and gold layers Driving … OneLake environments What we're looking for Strong commercial experience with Databricks in production environments Excellent data modelling and architecture skills Expertise in Delta Lake and Medallion design principles Azure Data Factory experience, including parameterised and environment-driven pipelines Strong SQL skills plus Python and/or Scala ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

trading decisions, risk management, and market analytics.Key Responsibilities Data Architecture: Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for structured/unstructured energy market data.Trading Analytics: Partner with Quants, Risk Managers, and Traders to implement models, curve generation, position tracking … trading data types (Reuters, Bloomberg, Trayport, weather, ETRM).Databricks Mastery: Hands-on experience with production workloads on Databricks (Azure/AWS), PySpark, Spark SQL, Delta Lake, Unity Catalog, Python, SQL, orchestration tools (Airflow, Workflows, ADF), and streaming (Kafka/Event Hubs).Professional Skills: Strong consultative mindset, excellent communication ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. ...