1 to 25 of 28 Delta Lake Jobs in the UK

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

considering: We are looking for an experienced Data Architect with strong expertise in Azure Databricks, Azure Data Factory, Azure Data Lake, Delta Lake, data modelling, data governance, and cloud data architecture, along with domain experience in London Market Insurance. The candidate will be responsible for defining … time. Your Role: Define end-to-end data architecture for Azure-based data platforms using Azure Databricks, Azure Data Factory, ADLS Gen2, Delta Lake, Azure Synapse, and Power BI. Design scalable and secure lakehouse architecture using bronze, silver, and gold data layers. Lead architecture and design for data ...

optimization of scalable data platforms and analytics solutions that support global data strategy. Architect and implement robust, scalable, metadata-driven pipelines in Databricks using Delta Lake, Unity Catalog, and declarative workflows. Oversee the development and maintenance of Azure SQL database objects using advanced T-SQL for data quality … enhance platform capabilities Essential Knowledge & Experience Extensive hands-on experience (6+ years) in cloud-native data engineering, including 3+ years with Databricks and Delta Lake in Azure, combined with good (2+ years) in a technical leadership or architecture role. Deep practical experience of the Databricks ecosystem including Delta ...

solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … drive engineering best practices Skills & Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms ...

/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure data security, governance, and compliance (access control, encryption, lifecycle) Provide technical leadership, mentoring and design reviews Collaborate with stakeholders … analysis, profiling and insight delivery Cloud & Platforms Azure: Data Factory, Synapse, Databricks, ADLS Gen2, Event Hub, Purview AWS: S3, Glue, Athena, Kinesis, Lambda, EMR, Lake Formation Databricks: Delta Lake, Unity Catalog, Structured Streaming, MLflow SQL Server stack (SSIS, SSRS) Engineering Practices CI/CD for data platforms ...

Databricks (Unity Catalog) Build scalable lakehouse and warehouse solutions using Databricks SQL and Fabric SQL endpoints Develop robust data pipelines with Databricks Workflows, Delta Live Tables (DLT), and Fabric Data Factory Define and implement cross-platform data strategies using Delta Lake to minimise duplication Establish data governance … Databricks (Unity Catalog) architecture Proven experience building modern lakehouse and data warehouse platforms Hands-on experience with: Databricks SQL & Fabric SQL Databricks Workflows & Delta Live Tables (DLT) Fabric Data Factory Deep knowledge of Delta Lake and cross-platform data strategies Solid experience in data governance, including lineage ...

real-time insights. Key Responsibilities Architecture & Solution Design Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures … leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata ...

timely, reliable and well-governed data across the organisation. You’ll: Design and develop scalable batch and streaming data pipelines using Databricks, Spark and Delta Lake Implement medallion architecture (bronze/silver/gold) to ensure data quality, lineage and efficient consumption Optimise Spark jobs and workflows … meet near real-time monitoring SLAs Build and maintain streaming pipelines using Structured Streaming and Delta Live Tables Ensure strong data governance , including schema enforcement, validation rules and ACID compliance Orchestrate and automate pipelines using Databricks Jobs, APIs or cloud-native schedulers Collaborate with data scientists, analysts and environmental ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

retry mechanisms. Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing. Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault. Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling … reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications. Experience with entity resolution, record matching platforms, and third-party enrichment services. Experience with Delta Lake and Change Data Feed (CDF) patterns. Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation. Experience ...

Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure SQL, Key Vault) as they relate ...

data platform aligns with the broader enterprise technology landscape. Skills and Experience Core Tech Stack Data Platform: Expert‐level knowledge of Databricks (Notebooks, Delta Lake, SQL endpoints). ETL/ELT: Proven experience building and managing data pipelines using tools like Azure Data Factory, dbt, or similar. ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

production experience is not required. A collaborative mindset and willingness to learn and communicate openly within the team. Nice to have Exposure to data lake concepts or formats such as Delta Lake or Iceberg. Any experience with search, graph, or large-scale analytical databases. Interest ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

data engineering. 2+ years experience as data engineer on AI/ML project with Python Strong experience using Databricks, including Jobs, Asset Bundles, Delta Lake, and MLflow, as well as Azure data engineering tools such as Azure Data Factory and Azure Data Lake Storage (ADLS). Solid ...

Databricks and Apache Spark. Build and optimize data workflows for batch and real-time processing. Transform and process large datasets using PySpark, SQL, and Delta Lake. Implement data ingestion from multiple sources (APIs, streaming, databases). Your Skills: Python proficiency. AWS : Cloud formation , Service Catalog , IAM , S3 , Glue , Lambda ...

solutions Designing scalable data and AI pipelines Implementing MLOps and LLMOps best practices We're interested in people with experience in: Python Databricks (PySpark, Delta Lake, Unity Catalog, MLflow, Mosaic AI) LLMs, GenAI and AI Agents MLOps and production ML systems CI/CD and DevOps ...

Technical Lead, Lead Data Engineer, or Technical Delivery Lead position. Strong hands-on expertise with Python. Extensive experience with Databricks, including Workflows, Notebooks, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and optimising large-scale ETL/ELT pipelines. Strong understanding of modern ...

Views, SLT, or equivalent SAP-native frameworks. Design and implement Medallion-aligned data pipelines (Bronze/Silver/Gold) in our cloud data lake or lakehouse (Azure/AWS/GCP/Databricks/Snowflake). Assess and remediate the current third-party implementation , identifying blockers and accelerating delivery. … strategy , including prioritisation of modules (FI/CO/MM/SD/PP etc.) required for December cutover. Build scalable ingestion pipelines using Delta Lake/Parquet and modern orchestration tools (ADF, dbx, Airflow, etc.). Collaborate with SAP functional teams to validate business logic, deltas ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn, and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...