1 to 25 of 127 Delta Lake Jobs in London

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

build, test and deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL. Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis … major cloud platform (Azure, AWS, or GCP). Strong experience designing and building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologies Proficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability. ...

build of large-scale data platforms using Azure Databricks and modern cloud technologies Implement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principles Enhance performance, manage cloud costs and ensure platform reliability for structured streaming workloads Define data governance, security and quality … optimization, data skew mitigation and query tuning Strong programming skills in Python with knowledge of modern software design principles Practical experience with structured streaming, Delta Lake and Delta Live Tables Proven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

Asset Bundles. Key Responsibilities Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design and manage Delta Tables, including … workflows. Package and deploy Databricks resources across environments using Databricks Asset Bundles and GitLab-based CI/CD pipelines. Integrate Databricks with Azure Data Lake Storage and Azure Data Factory for secure, reliable data processing. Optimize Spark workloads, clusters, jobs, and storage patterns for performance, scalability, reliability, and cost ...

GitHub Actions and Terraform setup for deployments Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture Work closely with analysts … they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows Tech stack Azure Databricks (Spark, Delta Lake) SQL Server, Confluent Kafka Scala, SQL, dbt Git, GitHub Actions, Terraform Sigma (BI) Must-have experience Commercial experience as a Data Engineer ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

maintain enterprise-grade ETL/ELT pipelines supporting ingestion, transformation, CDC, validation, and data delivery across multiple business functions. Develop and support Lakeflow pipelines, Delta Lake implementations, and modern data engineering frameworks. Drive best practices across performance optimisation, security, governance, monitoring, and platform reliability. Collaborate with business, data … development skills. Expertise building and supporting large-scale ETL/ELT pipelines and cloud-based data platforms. Strong understanding of Delta Lake, Unity Catalog, data governance principles, and medallion architecture. Experience integrating REST APIs, JSON data sources, Azure Data Factory, ADLS, Key Vault, and other Azure services. Hands ...

building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline Implementing governance … with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data quality and governance experience, ideally including Azure Purview Git, Agile delivery and a collaborative approach to code ...

thrive. Your day at NTT DATA Primary Responsibilities Client Engagement & Delivery Data Pipeline Development (Batch and Streaming) Databricks & Lakehouse Architectures Data Modelling & Optimisation (Delta Lake, Medallion architecture) Collaboration & Best Practices Quality, Governance & Security Business Relationships Solution Architects Data Engineers, Developers, ML Engineers, and Analysts Client stakeholders … mentoring teams of engineers to deliver high-quality scalable data solutions. Technical Expertise Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow). Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent. Strong SQL and Python ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

PySpark. Develop ETL/ELT and data transformation workflows using Python and SQL. Create and maintain Databricks notebooks, jobs, workflows and clusters. Work with Delta Lake and lakehouse architectures. Monitor, troubleshoot and optimise pipeline performance. Maintain high standards of data quality and reliability. Automate processes and create reusable … years’ Data Engineering experience. Strong hands‐on Databricks, PySpark, Python and SQL skills. Experience building and optimising ETL/ELT data pipelines. Knowledge of Delta Lake and data warehousing. Experience with at least one of AWS, Azure or GCP. Strong troubleshooting and problem-solving skills. A Computer Science ...

DoArchitect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments. Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate. Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration. What We're Looking ForProven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade data ...

seeking a Senior Data Engineer to design and optimize scalable data pipelines using Databricks, PySpark, Python and SQL, gaining hands-on experience with Delta Lake and lakehouse architectures. You will build ETL/ELT workflows, monitor performance, mentor juniors and contribute to cloud data platforms across AWS, Azure ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

will be a senior member of the engineering team, responsible for building large-scale Azure Data platforms, data models and working on the Delta Lake that underpins their Databricks AI platforms. This is a fully remote role, but you also have the option to work from multiple … Need: Strong experience as a Senior Azure Data Engineer An expert in Azure Databricks and Medallion Architecture Strong skills in PySpark, Python, Delta Lake and Unity Catalog Knowledge of Kimball data modelling, such as star schema, snowflake, etc. Strong SQL skills A flexible approach to getting things done ...

regularly observe cloud hardware, network, and operating system faults, and our software must gracefully shield our customers from any of the above. The Delta DML team owns the core write-path operations for Delta Lake, the open-source storage layer behind the Databricks Lakehouse. Our mission … flowing through our platform. We drive performance innovations like Low Shuffle Merge and Deletion Vectors and actively contribute to open source efforts to unify Delta and Iceberg formats. We are seeking a highly skilled and experienced Senior Staff Software Engineer to join our backend team. In this role ...

reporting. You’ll work on: Building and orchestrating ingestion pipelines for complex financial data Designing scalable data models across relational, NoSQL, blob and Delta Lake architectures Developing production APIs and data services for the wider SaaS platform Transforming and reconciling inconsistent structured and unstructured data Improving data quality … Care about data quality, clean engineering and genuine ownership Have experience building SaaS products or other production platforms Experience with Azure, Node.js, data warehousing, Delta Lake or vector databases would be valuable. You don’t need previous private-markets experience. What matters is strong engineering judgement ...

will be a senior member of the engineering team, responsible for building large-scale Azure Data platforms, data models and working on the Delta Lake that underpins their Databricks AI platforms. This is a fully remote role, but you also have the option to work from multiple … Need: Strong experience as a Senior Azure Data Engineer An expert in Azure Databricks and Medallion Architecture Strong skills in PySpark, Python, Delta Lake and Unity Catalog Knowledge of Kimball data modelling, such as star schema, snowflake, etc. A flexible approach to getting things done with a focus ...