1 to 25 of 29 Remote/Hybrid Delta Lake Jobs in the UK

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

frameworks and modern data engineering solutions Provide technical leadership and architectural guidance across data platform initiatives Design and govern Lakehouse architectures leveraging Databricks and Delta Lake Manage sprint planning, backlog prioritisation and delivery tracking within Agile teams Identify and mitigate delivery risks, dependencies and technical challenges Troubleshoot complex … Technical Lead, Engineering Lead, or similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture ...

/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure data security, governance, and compliance (access control, encryption, lifecycle) Provide technical leadership, mentoring and design reviews Collaborate with stakeholders … analysis, profiling and insight delivery Cloud & Platforms Azure: Data Factory, Synapse, Databricks, ADLS Gen2, Event Hub, Purview AWS: S3, Glue, Athena, Kinesis, Lambda, EMR, Lake Formation Databricks: Delta Lake, Unity Catalog, Structured Streaming, MLflow SQL Server stack (SSIS, SSRS) Engineering Practices CI/CD for data platforms ...

Build and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF). Develop Change Data Capture (CDC) solutions to support incremental and delta data loads. Create data transformation and cleansing processes using PySpark and Spark DataFrames. Develop Python-based utilities and automation scripts to support migration … optimise data models, SQL queries, stored procedures and analytical views within Azure Synapse Analytics. Manage data ingestion, transformation and storage across Azure Data Lake Storage (ADLS) and Azure SQL environments. Support processing of structured and semi-structured data formats, including Delta Lake and JSON. Collaborate with stakeholders ...

global environment. Description Design, build and maintain scalable data pipelines using Python and Databricks Develop and optimise ETL/ELT processes leveraging Spark and Delta Lake Create robust data models and data architectures within modern Lakehouse environments Implement and manage data workflows across the Databricks ecosystem Ensure data … Capital Markets, Insurance, or other highly regulated enterprise environments. Strong hands-on expertise in Python Proven experience with Databricks, including notebooks, workflows, jobs and Delta Lake Strong knowledge of Apache Spark/PySpark Experience designing and delivering large-scale ETL/ELT pipelines Advanced SQL skills and experience ...

dimensional modelling) for consistent implementation across data solutionsEvaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patternsDesign technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)Create and maintain architecture documentation, including data flow diagrams, architecture … recommend new technologies and tools that enhance Arriva's data and BI capabilitiesStay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutionsConduct proofs-of-concept and technical assessments to validate new technologies before adoptionMonitor ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

Data Engineer will design, build, support and operate data pipelines and core data platform technologies within the organisation's Azure-based analytics and lake house environment. The role is hands-on and implementation-focused, working within established architectural patterns to deliver reliable, secure and well-governed data for analytics … Responsibilities Develop and maintain data pipelines using Azure Databricks (Python, PySpark and SQL) Implement ingestion and transformation across Bronze, Silver and Gold layers using Delta Lake Support and operate Azure Databricks and Azure Synapse platforms, including performance monitoring and configuration Build Gold-layer datasets optimised for Power ...

governance controls and operational guardrails Technical Leadership & Standards Serve as the senior technical authority for the Databricks platform architecture Establish engineering best practices covering Delta Lake design, workload orchestration, compute optimization, and cost management Provide technical assurance across platform implementation activities delivered by internal teams and external partners … communication, stakeholder engagement, and consulting capabilities Experience translating business requirements into strategic platform architecture solutions Technical Skills Deep expertise in Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, Workflows, and cluster management Strong experience designing cloud-native data platforms on AWS Hands‐on knowledge of AWS services including ...

risks, exceptions, and technical dependencies raised through governance processes Enterprise & Solution Architecture Provide architecture assurance for Databricks Lakehouse implementations, including data ingestion, medallion architecture, Delta Lake, Unity Catalog, and compute strategies Validate integration patterns across Databricks, cloud platforms, enterprise applications, and data ecosystems Define and maintain reference architectures … delivery requirements Exceptional stakeholder engagement and influencing skills across technical and business audiences Technical Skills Strong hands‐on knowledge of Databricks including Lakehouse Architecture, Delta Lake, Unity Catalog, and platform governance Experience with cloud platforms supporting Databricks environments, including AWS and/or Azure Solid understanding of enterprise ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

retry mechanisms. Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing. Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault. Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling … reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications. Experience with entity resolution, record matching platforms, and third-party enrichment services. Experience with Delta Lake and Change Data Feed (CDF) patterns. Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation. Experience ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

environment strategy. Design and govern data integration patterns for sharing data with external clients and third parties, including secure, scalable approaches using Databricks Delta Sharing, APIs, and file-based exchange where appropriate. Lead the design and adoption of Master Data Management (MDM), including entity definitions, golden-record strategy, match … merge rules, and stewardship processes. Establish standards and practices for optimizing Databricks performance and cost — cluster and compute strategy, Delta table management, caching, and workload right-sizing. Define data modelling standards (conceptual, logical, physical) across analytical and operational domains. Collaborate with data governance to embed data quality, lineage, cataloguing ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

Databricks and Apache Spark. Build and optimize data workflows for batch and real-time processing. Transform and process large datasets using PySpark, SQL, and Delta Lake. Implement data ingestion from multiple sources (APIs, streaming, databases). Your Skills: Python proficiency. AWS : Cloud formation , Service Catalog , IAM , S3 , Glue , Lambda ...

solutions end-to-end, with a focus on Lakehouse Architecture and ML/AI-driven analytics. Design scalable data models and pipelines using Spark, Delta Lake, and PySpark. Lead client engagements, shaping data strategies and best‐practice architectures. Win business through presenting solutions and advising potential clients. Optimise ...

solutions Designing scalable data and AI pipelines Implementing MLOps and LLMOps best practices We're interested in people with experience in: Python Databricks (PySpark, Delta Lake, Unity Catalog, MLflow, Mosaic AI) LLMs, GenAI and AI Agents MLOps and production ML systems CI/CD and DevOps ...

into technical solutions Requirements: Enterprise data warehouse architecture and design Microsoft Fabric, Azure Data Platform technologies and modern Lakehouse architectures Databricks, Unity Catalog and Delta Lake PySpark, Databricks, SQL Medallion Architecture implementation Data modelling approaches including Star Schema and Data Vault Data governance, security and access management ...

legacy operational platforms. Experience with regulatory, client or operational reporting. Experience supporting data migration, legacy system modernisation or platform replacement. Experience with Databricks, Delta Lake, Spark SQL or PySpark. Experience defining validation frameworks, data quality dashboards or metadata-driven rule engines. Experience working in consultancy or client-facing ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...