1 to 25 of 44 Delta Lake Jobs in the UK excluding London

optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data platforms. Lead … deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver ...

drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast, trusted data at their fingertips. … governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop robust Bronze, Silver ...

platforms such as Azure, AWS or GCP* Define lakehouse standards, data modelling approaches and governance frameworks* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies* Establish security, access control, data quality and compliance standards* Optimise performance, scalability and cloud cost management* Lead … experience* Strong experience as a senior or lead data engineer building and scaling production data platforms* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows* Python and SQL proficiency* Infrastructure as code experience using Terraform, Bicep or CloudFormation* Cloud data platform experience, ideally Azure ...

platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead … experience * Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

engineering challenges, and driving best practice across the platform. What you'll be doing Designing, building and optimising Databricks pipelines and jobs Creating scalable Delta Lake solutions using Medallion architecture Developing and maintaining Azure Data Factory pipelines Owning data models across bronze, silver and gold layers Driving … OneLake environments What we're looking for Strong commercial experience with Databricks in production environments Excellent data modelling and architecture skills Expertise in Delta Lake and Medallion design principles Azure Data Factory experience, including parameterised and environment-driven pipelines Strong SQL skills plus Python and/or Scala ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. ...

managing and reporting control issues with transparency.**Skills & Experience****Essential**Proven in-depth commercial experience in the following key areas:* Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning.* Apache Spark & Scala – High-performance distributed data processing and optimisation.* Cloud Data Engineering (AWS) – Cloud-native data platforms … core business functions* Demonstrated leadership, project management, and development skills* Relationship and consensus building skills**Preferred*** Kafka/Event-Driven Architecture* Apache Iceberg/Delta Lake* Python* AI/ML & LLM integration* Financial Services/Capital Markets**Education:*** Bachelor’s degree/University degree or equivalent experience* Master ...

scalable ETL/ELT data pipelines. Microsoft Fabric, including Data Factory, Dataflow Gen2 and Fabric Notebooks. Databricks Workflows and Lakehouse technologies. PySpark, SQL and Delta Lake environments. Data ingestion, transformation and orchestration processes. Data governance, security and quality initiatives. Enterprise-wide analytics and reporting solutions, including Power … data transformation best practices. Experience building and supporting production-grade data pipelines. Hands-on experience with Microsoft Fabric. Experience with Databricks and Delta Lake technologies. Understanding of Lakehouse architecture and modern data platforms. Experience delivering data solutions that support analytics and business reporting. Interested? This is a rare ...

growth. Skills, Knowledge & Expertise Experience and knowledge: Experience of designing and implementing an Azure analytical platform including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Delta Lake, Unity Catalog, PySpark, SQL, Python, and Azure DevOps. Experience of working with senior stakeholders to articulate the strategy, objectives ...

Skills Strong experience designing and delivering modern data platforms using Microsoft Fabric, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL and Data Lake technologies . Proven experience implementing Lakehouse, Data Warehouse and Medallion Architecture patterns, including data modelling, governance and performance optimization . Expertise in building scalable … Python, Spark and PySpark for data engineering, analytics and automation workloads. Experience with Azure Databricks and/or Microsoft Fabric engineering workloads, including notebooks, Delta Lake, data transformation, optimization and operationalization of data solutions. An understanding of data governance, security, lineage and data quality, using tools and frameworks ...

Strong Azure Data Factory (ADF) and Azure Databricks experience. Experience building ETL/ELT pipelines within Azure Lakehouse environments. Knowledge of Medallion Architecture and Delta Lake. Strong SQL and Python skills. Experience with Power BI datasets and data modelling. Familiarity with Azure DevOps, Git, CI/CD and Infrastructure ...

communicate complex concepts in a clear, concise manner Languages: Terraform, Python, SQL, PowerShell. CI/CD: Azure DevOps pipelines for data platform deployments. Databricks: Delta Lake, Unity Catalog, Workflows, Notebooks, Clusters, MLflow. Azure: ADLS Gen2, Azure Data Factory, Event Hubs/Kafka, Key Vault, Functions, Synapse. AI Foundry … Service Plan Deployment and Configuration, Integration with Azure bot Services MS Fabric: Tenant Management, Workspace Private Links, Deployment, Capacity Management, Workspace Management, One Lake, Data Mirroring, Power BI AI Foundry : Model management, Hosted Agents Deployment and Management , Project Configuration (Connections and Governance ) and lifecycle management. Inclusion & Diversity We value ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

pipeline development, and large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security … Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta Lake file formats.J.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most prominent corporations ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

enable timely, reliable and well-governed data across the organisation.You’ll:Design and develop scalable batch and streaming data pipelines using Databricks, Spark and Delta LakeImplement medallion architecture (bronze/silver/gold) to ensure data quality, lineage and efficient consumptionOptimise Spark jobs and workflows to meet near real … time monitoring SLAsBuild and maintain streaming pipelines using Structured Streaming and Delta Live TablesEnsure strong data governance, including schema enforcement, validation rules and ACID complianceOrchestrate and automate pipelines using Databricks Jobs, APIs or cloud-native schedulersCollaborate with data scientists, analysts and environmental specialists to deliver high-quality, well-documented ...

environments, ideally including Microsoft Fabric, Azure Databricks, or both. Strong understanding of Azure services and operational dependencies, including workspaces, capacities, lakehouses, pipelines, clusters, jobs, Delta Lake, Unity Catalog, OneLake, Key Vault, Azure Monitor, Log Analytics, Entra ID, managed identities, secure networking, and private endpoint architectures. Proven experience ...

data concepts, including storage, processing, and modelling. Core requirements include:Databricks - hands-on experience building and managing workloads on the Databricks platform, including Delta Lake, Unity Catalog, and Declarative Automation BundlesPySpark - proficiency in writing and optimising PySpark code for data transformation and processingAzure - solid experience working within … Azure ecosystem, including services such as Azure Data Factory, Azure Data Lake Storage, and Azure DevOpsAzure SQL Server - experience with Azure SQL Server is important, as it forms the foundation of our existing data warehouse. You will be working across both our SQL Server environment and our Databricks platform ...

improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration patterns for REST … Azure Databricks implementation and troubleshooting Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration ...

sensitivity considerations, secure handling of inputs/outputs, and adherence to resiliency and security expectationsPreferred qualifications, capabilities, and skillsExperience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworksFamiliarity with AWS-native data services such as Glue, Lake Formation, Redshift, or AthenaExposure ...

secure handling of inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena ...