1 to 25 of 37 Delta Lake Jobs in England

pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED... Please either ...

reference data in relational databases and Databricks lakehouse tables. 5. Databricks Integration — Leverage Databricks as a core data platform — Unity Catalog for data governance, Delta Lake for reliable storage, Databricks Vector Search for managed embeddings, and Spark/SQL for large-scale data processing and feature engineering. 6. … Understanding of ontology design and entity resolution for financial entities (counterparties, instruments, legal entities). Structured Data & Databricks (must-have) ● Production experience with Databricks — Delta Lake, Unity Catalog, Spark SQL, and ideally Databricks Vector Search or Databricks Feature Serving. ● Experience with text-to-SQL approaches — using LLMs ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … streaming platforms (e.g. Event Hubs, Kafka), and legacy databases into ADLS Gen2 and OneLake Develop structured data models aligned to Lakehouse, Medallion Architecture, and Delta Lake patterns Continuously profile, debug, and optimise Spark jobs, SQL queries, and Python scripts for maximum performance and cost-efficiency at scale Champion ...

work at home 100% of the time. Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance … continuous improvement. Your Skills: Expert-level hands-on experience with Azure Data Services and Databricks in enterprise, highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing ...

zone into the Bronze layer, working in partnership with the Development and Application Management team.Design and optimise ETL/ELT pipeline frameworks using Databricks, Delta Lake and Azure Data Factory. * Ensure Silver and Gold layer data products are fit for purpose for analytics, pricing, AI and ML model … scalability. Experience in regulated financial services, ensuring compliance with industry standards. Deep expertise in data modelling - conceptual, logical and physical. Data warehousing and data lake architecture for high-performance analytics. ETL/ELT pipeline development and optimisation to support large-scale data processing. Data integration across structured and unstructured ...

elevating engineering capability across engagements. What you'll do Lead the engineering, optimisation and governance of Azure-based data platforms, spanning ADF, Data Lake, Azure Functions, Key Vault, Databricks, Delta Lake, PySpark and Unity Catalog . Collaborate closely with insurance domain teams- Underwriting, Actuarial, Delegated Authority, Bordereaux … efficient operation. What you'll bring Expert-level hands-on experience with Azure data services and Databricks in enterprise-scale environments. Strong knowledge of Delta Lake , Medallion architecture, distributed compute and Lakehouse engineering patterns. Advanced Python, PySpark and Spark SQL skills, including deploying data workloads through Azure DevOps ...

real-time insights. Key Responsibilities Architecture & Solution Design Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures … leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata ...

Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud platforms and enterprise data sources … deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern data architecture ...

readiness, and regulatory compliance. Your responsibilities: Design and govern enterprise data architectures using Microsoft Foundry, Azure Data Factory, ADLS, Synapse, and Databricks (Spark & Delta Lake). Lead data modelling initiatives including star schema and lakehouse patterns. Architect end-to-end ETL/ELT pipelines and integration strategies. Ensure … compliance in regulated environments. Your ProfileEssential skills/knowledge/experience: Strong experience with Microsoft Foundry, Azure (Data Factory, ADLS, Synapse) and Databricks (Spark, Delta Lake). Proficient in Python and SQL. Proven track record designing end-to-end data pipelines and architectures. Deep understanding of ETL/ ...

Data Board. Champion reuse, standardisation, and scalable design across all data products. Delivery Leadership (Azure & Databricks Lakehouse) Lead cross‐functional squads building ingestion pipelines, Delta Lake tables, and Power BI semantic models. Drive Agile delivery, manage backlogs, and lead sprint ceremonies. Ensure reliability, observability, performance, and smooth handover … Experience Technical Expertise Strong understanding of Azure data services (e.g., ADF, Synapse pipelines, EventHub/ServiceBus, Key Vault). Hands‐on experience with Databricks (Delta Lake, Unity Catalog, workflows, notebooks). Knowledge of Power BI modelling, semantic layers, DAX fundamentals, RLS/OLS. Familiarity with CI/ ...

Databricks, including target architecture, data models, security, and governance frameworks. Analyze existing Foundry pipelines, datasets, and ontologies, and redesign them into scalable Databricks (Delta Lake) ETL/ELT workflows using PySpark and SQL. Implement validation frameworks to ensure complete data accuracy, consistency, and lineage throughout the migration process. … business stakeholders to minimize migration risk, ensure smooth cutover, and maintain production stability. Skills Must have Advanced Databricks Experience. Deep knowledge of Databricks architecture, Delta Lake, job orchestration, cluster management, and performance tuning. Proven Migration Experience (Palantir → Databricks). Demonstrated experience leading or executing platform migrations, including pipeline ...

energy market data using Python, SQL and Spark. Support data flows ensuring generation and storage assets deliver the correct contracted schedules. Design and manage Delta Lake tables and SQL databases, applying strong data modelling and optimisation strategies. Develop monitoring dashboards and alerting to ensure transparency around pipeline health … similar production-focused role Strong Python (Pandas) and SQL skills Experience building and maintaining data pipelines Hands-on experience with Spark, Databricks or Delta Lake/Lakehouse architectures Familiarity with cloud data platforms, particularly within Azure environments Good understanding of data modelling, storage formats and performance optimisation Experience ...

DSFS data model and existing Netezza structures. Produce the EDW logical and physical data models, dimensional models for the Semantic Layer, and Data Lake zone architecture (Bronze/Silver/Gold). Define Enterprise Semantic Layer design, including pre-computed aggregations, dimensional models, and Tableau connectivity specifications. Lead source … data architecture, with at least 3 years leading enterprise data model design. Experience designing Lakehouse architecture: medallion architecture (Bronze/Silver/Gold), Delta Lake, or equivalent. Proficiency in dimensional modelling (Kimball methodology) and enterprise data modelling. Experience with Databricks (Unity Catalog, Delta Lake) or Snowflake ...

collaboration skills. Familiarity with Microsoft Azure tools such as Azure Data Factory, Power BI, Data Pipelines, and Data Lake. Experience with Azure Data Lake and Delta Lake design pattern implementation, including file hierarchy and low-level permissions management (desirable). Experience in building CI/CD pipelines ...

role involves leading the development of secure, reliable, and efficient data pipelines using Azure Synapse Analytics, Azure Data Factory, PySpark, and Azure Data Lake Storage. The engineer will work closely with cross functional teams to translate business requirements into technical solutions that support analytics, reporting, and data driven decision … data transformations within Azure Synapse Analytics. Implement incremental loads, CDC, and real-time data ingestion when needed. Architect and manage datasets in Azure Data Lake Storage (ADLS Gen2). Optimize data storage formats (Delta, Parquet, Avro, ORC) for performance and cost. Maintain and optimize Synapse SQL Pools, Dedicated ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation's data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

reliability Collaborate with analysts and AI engineers to deliver robust and reusable data products Manage data lakes and warehouses using formats such as Delta Lake and Parquet Implement best practices for data governance, performance, and security Continuously evaluate and adopt new technologies to evolve the organisation’s data … platform Provide technical guidance to junior engineers and contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable ...

Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

journeys. What You'll Bring Extensive experience designing and delivering end-to-end data platforms using Databricks , with a strong focus on Lakehouse Architecture , Delta Lake, and scalable data engineering patterns. A solid background in data analytics, consulting, or a similar technical discipline. Strong knowledge of structured … Kimball , and Data Vault . Hands-on experience exposing Lakehouse solutions to BI tools (e.g. Power BI , Tableau ). Proficiency in Databricks Workflows , Delta Live Tables , and Spark for batch and streaming pipelines. Strong programming capability in Python , Scala , and/or SQL . Practical experience with at least ...

equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics, Azure Data Lake Storage; infrastructure as code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs ...

target mapping, data quality rule setup for the target data model’s critical data elements (CDE). Database Optimization: Optimize Cloud databases such as Delta Lake or One Lake for efficiency, scalability, and faster access using techniques like indexing and partitioning. Data Governance: Work with Federated Governance ...

Bronze/Silver/Gold). Developing scalable models and Power BI semantic layers for enterprise reporting. Enhancing and supporting components across Azure Data Lake, ADF, Synapse, Azure SQL and related Microsoft stack tools. Applying engineering discipline across testing, code management, CI/CD and performance optimisation. Working closely … alignment with architectural standards. What You'll Need to Succeed Strong hands-on engineering experience within Microsoft Fabric (Lakehouse, Pipelines, Dataflows Gen2, Notebooks, Direct Lake). Previous experience with Azure Databricks and Delta Lake engineering. Solid working knowledge of the Microsoft data stack: Azure Data Factory, ADLS ...

local enough to do this without relocating. What you’ll be doing Build, enhance and maintain data pipelines using Azure Databricks , Data Factory , and Delta Lake Develop and optimise Lakehouse components and cloud-based data flows Create robust data models to support analytics, MI and downstream reporting Assist ...