1 to 25 of 62 Delta Lake Jobs in the UK

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

Build and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF). Develop Change Data Capture (CDC) solutions to support incremental and delta data loads. Create data transformation and cleansing processes using PySpark and Spark DataFrames. Develop Python-based utilities and automation scripts to support migration … optimise data models, SQL queries, stored procedures and analytical views within Azure Synapse Analytics. Manage data ingestion, transformation and storage across Azure Data Lake Storage (ADLS) and Azure SQL environments. Support processing of structured and semi-structured data formats, including Delta Lake and JSON. Collaborate with stakeholders ...

scalable Lakehouse solutions supporting critical business data. What you'll be doing Leading Azure Databricks architecture Designing enterprise Lakehouse solutions Building scalable Spark and Delta Lake pipelines Designing streaming and batch data platforms Driving engineering best practice Mentoring engineers Working closely with senior business stakeholders Defining future-state … data architecture Technical Environment Azure Databricks Apache Spark/PySpark Delta Lake Azure Data Factory Azure Data Lake Gen2 Azure Event Hubs Azure Synapse Python SQL Terraform CI/CD Medallion Architecture We're looking for Expert Databricks experience Strong Azure Data Platform experience Enterprise Data Architecture ...

Data Engineer will design, build, support and operate data pipelines and core data platform technologies within the organisation's Azure-based analytics and lake house environment. The role is hands-on and implementation-focused, working within established architectural patterns to deliver reliable, secure and well-governed data for analytics … Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including performance monitoring and configuration Build Gold-layer datasets optimised for Power ...

Architect and lead the end-to-end delivery of Databricks-based data platforms across Azure environments. Design secure, scalable data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services, applying AI-enabled engineering approaches where appropriate. Establish and promote engineering best practices across … approved AI tools and automation to enhance delivery and collaboration. What We're Looking For Proven expertise with the Databricks platform, including Unity Catalog, Delta Lake, orchestration, and enterprise-scale Azure deployments. Strong technical skills in Python, PySpark, and distributed data processing frameworks, with experience building production-grade ...

enterprise-grade business insights. Key Responsibilities Data Architecture & CDL Governance: Design, implement, and govern the enterprise Common Data Layer (CDL) on Azure Databricks using Delta Lake and L0–L1–L2 architectures. Establish data quality frameworks, schema evolution standards, and deduplication logic. Pipeline & Integration Engineering: Oversee the creation … Databricks data assets into robust, secure, and intuitive Power BI dashboards. Required Experience & Skills Technical Expertise Cloud Data Platforms: Advanced knowledge of Azure Databricks, Delta Lake, Azure Data Factory (ADF), and SQL querying. SAP Ecosystem: Deep data discovery skills within SAP S/4HANA, SAP BW/HANA ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

managing and reporting control issues with transparency.**Skills & Experience****Essential**Proven in-depth commercial experience in the following key areas:* Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning.* Apache Spark & Scala – High-performance distributed data processing and optimisation.* Cloud Data Engineering (AWS) – Cloud-native data platforms … core business functions* Demonstrated leadership, project management, and development skills* Relationship and consensus building skills**Preferred*** Kafka/Event-Driven Architecture* Apache Iceberg/Delta Lake* Python* AI/ML & LLM integration* Financial Services/Capital Markets**Education:*** Bachelor’s degree/University degree or equivalent experience* Master ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

enriching, and aggregating data. Ensure data consistency and accuracy throughout the data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need ...

retry mechanisms. Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing. Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault. Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling … reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications. Experience with entity resolution, record matching platforms, and third-party enrichment services. Experience with Delta Lake and Change Data Feed (CDF) patterns. Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation. Experience ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

files, and event streams Supporting master data management, data quality rules, and golden record definition Working hands-on with ADLS, Azure SQL, Event Hubs, Delta Lake, and OneLake Designing integration patterns and REST APIs to support near real-time data flows Contributing to infrastructure-as-code and environment ...

production experience is not required. A collaborative mindset and willingness to learn and communicate openly within the team. Nice to have Exposure to data lake concepts or formats such as Delta Lake or Iceberg. Any experience with search, graph, or large-scale analytical databases. Interest ...

enhance solution quality while applying sound engineering judgement and responsible AI practices. What You'll Do Design and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services. Build real-time, event-driven solutions using technologies such as Kafka and Azure Event ...

extraction through reconciliation) Microsoft Fabric experience is a bonus, not a requirement - strong Azure Databricks/Synapse background welcomed Nice to have: Microsoft Purview, Delta Lake/medallion architecture, Key Vault/Managed Identities, Azure DE Associate or Databricks certification. ...

delivering large scale Data science or AI Projects Strong knowledge of LLMs (GPT, Llama, Claude, Mistral) and their reasoning capabilities Deep Databricks experience (DLT, Delta Lake, Unity Catalog governance) Solid grasp of streaming technologies (Spark Structured Streaming, Autoloader) Python, SQL, or Scala proficiency Experience with ETL/ ...

data fidelity. Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

model lifecycle: data prep, experimentation, deployment, and monitoring.* Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS.* Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing.* Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering.* Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering.* Experience with Databricks, Spark, and Delta Lake.* Strong SQL skills with experience working on large datasets.* Experience with AWS, including S3 and Lambda.* Familiarity with Redshift, Snowflake, or other cloud ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

solutions Designing scalable data and AI pipelines Implementing MLOps and LLMOps best practices We're interested in people with experience in: Python Databricks (PySpark, Delta Lake, Unity Catalog, MLflow, Mosaic AI) LLMs, GenAI and AI Agents MLOps and production ML systems CI/CD and DevOps ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. We take great ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs of all of our employees. ...