1 to 25 of 90 Delta Lake Jobs in the UK

optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data platforms. Lead … deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

cost. Implement compute policies, autoscaling, quotas, budgets and operational guardrails. Troubleshoot production issues and continuously improve platform performance and reliability. Optimise Databricks SQL, Delta Lake and workload performance. FinOps & Governance Establish practical FinOps processes covering ownership, cost centres, projects, applications and teams. Implement tagging standards and automated compliance … data scientists, engineers, architects and business stakeholders to drive adoption. Data Engineering & Integration Build scalable ingestion and transformation solutions using Python, PySpark, SQL and Delta Lake . Deliver batch, incremental and CDC processing, schema evolution, reconciliation, data quality and error handling. Integrate REST APIs, SaaS platforms, databases, files ...

platforms such as Azure, AWS or GCP Define lakehouse standards, data modelling approaches and governance frameworks Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies Establish security, access control, data quality and compliance standards Optimise performance, scalability and cloud cost management Lead … experience Strong experience as a senior or lead data engineer building and scaling production data platforms Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows Python and SQL proficiency Infrastructure as code experience using Terraform, Bicep or CloudFormation Cloud data platform experience, ideally Azure ...

platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead … experience * Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure ...

data from internal systems and third-party platforms including Salesforce, Sopra Banking Services, CACI and Experian. Engineer Modern Data Architectures - Design Lakehouse, Warehouse and Delta Lake solutions using medallion architectures to support enterprise analytics and reporting. Deliver Analytics-Ready Data Assets - Build trusted, well-modelled datasets and semantic … successfully built, enhanced or operated cloud-based data platforms, ideally within Azure environments. Data Architecture Knowledge - You understand data modelling, ELT/ETL design, Delta Lake principles, data quality frameworks and modern engineering best practices. Cloud & DevOps Capability - You have experience with Azure services, automation, CI/ ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

datasets, performant ETL/ELT pipelines, and well-governed data products. Job responsibilities Design and implement batch and streaming data pipelines using Databricks (Spark), Delta Lake , and orchestrators (e.g., Workflows , Airflow, ADF). Develop and optimize Spark jobs (PySpark/Scala) and SQL transformations for performance, reliability … Databricks and/or Apache Spark in production. Strong coding skills in Python (PySpark) and SQL (Scala a plus). Hands-on experience with Delta Lake (MERGE/UPSERT patterns, schema evolution, partitioning, Z-ORDER, OPTIMIZE/VACUUM). Experience with orchestration and scheduling (Databricks Workflows, Airflow, Azure ...

enterprise environments 4+ years of hands‐on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

engineering challenges, and driving best practice across the platform. What you'll be doing Designing, building and optimising Databricks pipelines and jobs Creating scalable Delta Lake solutions using Medallion architecture Developing and maintaining Azure Data Factory pipelines Owning data models across bronze, silver and gold layers Driving … OneLake environments What we're looking for Strong commercial experience with Databricks in production environments Excellent data modelling and architecture skills Expertise in Delta Lake and Medallion design principles Azure Data Factory experience, including parameterised and environment-driven pipelines Strong SQL skills plus Python and/or Scala ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. ...

will be a senior member of the engineering team, responsible for building large-scale Azure Data platforms, data models and working on the Delta Lake that underpins their Databricks AI platforms. This is a fully remote role, but you also have the option to work from multiple … Need: Strong experience as a Senior Azure Data Engineer An expert in Azure Databricks and Medallion Architecture Strong skills in PySpark, Python, Delta Lake and Unity Catalog Knowledge of Kimball data modelling, such as star schema, snowflake, etc. Strong SQL skills A flexible approach to getting things done ...

managing and reporting control issues with transparency.**Skills & Experience****Essential**Proven in-depth commercial experience in the following key areas:* Databricks – Lakehouse architecture, Spark, Delta Lake, workflows and performance tuning.* Apache Spark & Scala – High-performance distributed data processing and optimisation.* Cloud Data Engineering (AWS) – Cloud-native data platforms … core business functions* Demonstrated leadership, project management, and development skills* Relationship and consensus building skills**Preferred*** Kafka/Event-Driven Architecture* Apache Iceberg/Delta Lake* Python* AI/ML & LLM integration* Financial Services/Capital Markets**Education:*** Bachelor’s degree/University degree or equivalent experience* Master ...

scalable ETL/ELT data pipelines. Microsoft Fabric, including Data Factory, Dataflow Gen2 and Fabric Notebooks. Databricks Workflows and Lakehouse technologies. PySpark, SQL and Delta Lake environments. Data ingestion, transformation and orchestration processes. Data governance, security and quality initiatives. Enterprise-wide analytics and reporting solutions, including Power … data transformation best practices. Experience building and supporting production-grade data pipelines. Hands-on experience with Microsoft Fabric. Experience with Databricks and Delta Lake technologies. Understanding of Lakehouse architecture and modern data platforms. Experience delivering data solutions that support analytics and business reporting. Interested? This is a rare ...

/ELT solutions and enterprise data pipelines. Experience with event-driven and messaging architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture ...

growth. Skills, Knowledge & Expertise Experience and knowledge: Experience of designing and implementing an Azure analytical platform including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Delta Lake, Unity Catalog, PySpark, SQL, Python, and Azure DevOps. Experience of working with senior stakeholders to articulate the strategy, objectives ...

Skills Strong experience designing and delivering modern data platforms using Microsoft Fabric, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL and Data Lake technologies . Proven experience implementing Lakehouse, Data Warehouse and Medallion Architecture patterns, including data modelling, governance and performance optimization . Expertise in building scalable … Python, Spark and PySpark for data engineering, analytics and automation workloads. Experience with Azure Databricks and/or Microsoft Fabric engineering workloads, including notebooks, Delta Lake, data transformation, optimization and operationalization of data solutions. An understanding of data governance, security, lineage and data quality, using tools and frameworks ...

sound judgment to tradeoffs around performance, cost, reliability, and maintainability. What you'll do Design, build, and operate production ETL pipelines on Databricks using Delta Live Tables and Spark, applying medallion architecture patterns across Finance, Payments, Product, and GTM domains. Own data modeling in different layers — defining governed, reusable … years of hands-on data engineering experience building and maintaining production pipelines and data models. Deep proficiency with Databricks or similar platforms — Delta Live Tables, Unity Catalog, Delta Lake, and PySpark or Spark SQL as primary tools. Strong SQL skills: complex transformations, incremental materialization, schema evolution ...

robust ETL/ELT solutions using Azure Data Factory and Microsoft Fabric. Develop high-performance data processing solutions using Python, PySpark, and SQL. Implement Delta Lake and Lakehouse architectures following industry best practices. Work with Azure Data Lake Storage (ADLS Gen2) to manage large-scale datasets. Optimise ...

Strong Azure Data Factory (ADF) and Azure Databricks experience. Experience building ETL/ELT pipelines within Azure Lakehouse environments. Knowledge of Medallion Architecture and Delta Lake. Strong SQL and Python skills. Experience with Power BI datasets and data modelling. Familiarity with Azure DevOps, Git, CI/CD and Infrastructure ...

workflows Excellent communication and collaboration skills in agile, cross-functional teams Ability to mentor junior engineers Nice to have: Experience with Databricks, PySpark, and Delta Lake Условия Competitive salary based on location, experience, and skills Comprehensive benefits package for employees and their families Continuous learning, structured development programs ...

communicate complex concepts in a clear, concise manner Languages: Terraform, Python, SQL, PowerShell. CI/CD: Azure DevOps pipelines for data platform deployments. Databricks: Delta Lake, Unity Catalog, Workflows, Notebooks, Clusters, MLflow. Azure: ADLS Gen2, Azure Data Factory, Event Hubs/Kafka, Key Vault, Functions, Synapse. AI Foundry … Service Plan Deployment and Configuration, Integration with Azure bot Services MS Fabric: Tenant Management, Workspace Private Links, Deployment, Capacity Management, Workspace Management, One Lake, Data Mirroring, Power BI AI Foundry : Model management, Hosted Agents Deployment and Management , Project Configuration (Connections and Governance ) and lifecycle management. Inclusion & Diversity We value ...