26 to 50 of 266 Delta Lake Jobs in England

cost and usage data platform on Databricks: ingesting Azure Cost Management exports, reservation and Savings Plan data, resource inventory and tagging metadata into Delta Lake, and modelling it into datasets that engineering and Finance both trust. Build and own the Power BI semantic models behind cost reporting … analysis Power BI - our primary surface for cost visibility: DAX, TMDL, composite and DirectQuery models, and XMLA endpoints for managing semantic models programmatically Databricks - Delta Lake and SQL warehouses, where our cost and usage data is processed, modelled and served KQL (Kusto Query Language) - querying Azure Monitor ...

seeking a Senior Data Engineer to design and optimize scalable data pipelines using Databricks, PySpark, Python and SQL, gaining hands-on experience with Delta Lake and lakehouse architectures. You will build ETL/ELT workflows, monitor performance, mentor juniors and contribute to cloud data platforms across AWS, Azure ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

Databricks Senior Data Engineer - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data modelling. Cloud platform ...

shared data assets Technical Leadership & Standards Serve as a senior technical authority for data product architecture across the program Define architectural best practices for Delta Lake design, metadata management, product lifecycle governance, and data quality controls Provide architecture assurance and design reviews for data products developed by internal … data contracts, service models, and consumer engagement frameworks Excellent stakeholder management, communication, and consulting skills Technical Skills Deep expertise in Databricks, including Lakehouse Architecture, Delta Lake, Unity Catalog, and Databricks SQL Strong understanding of data modelling, schema design, metadata management, and lineage tracking Experience implementing self-service data ...

risks, exceptions, and technical dependencies raised through governance processes Enterprise & Solution Architecture Provide architecture assurance for Databricks Lakehouse implementations, including data ingestion, medallion architecture, Delta Lake, Unity Catalog, and compute strategies Validate integration patterns across Databricks, cloud platforms, enterprise applications, and data ecosystems Define and maintain reference architectures … delivery requirements Exceptional stakeholder engagement and influencing skills across technical and business audiences Technical Skills Strong hands-on knowledge of Databricks including Lakehouse Architecture, Delta Lake, Unity Catalog, and platform governance Experience with cloud platforms supporting Databricks environments, including AWS and/or Azure Solid understanding of enterprise ...

regularly observe cloud hardware, network, and operating system faults, and our software must gracefully shield our customers from any of the above. The Delta DML team owns the core write-path operations for Delta Lake, the open-source storage layer behind the Databricks Lakehouse. Our mission … flowing through our platform. We drive performance innovations like Low Shuffle Merge and Deletion Vectors and actively contribute to open source efforts to unify Delta and Iceberg formats. We are seeking a highly skilled and experienced Senior Staff Software Engineer to join our backend team. In this role ...

reporting. You’ll work on: Building and orchestrating ingestion pipelines for complex financial data Designing scalable data models across relational, NoSQL, blob and Delta Lake architectures Developing production APIs and data services for the wider SaaS platform Transforming and reconciling inconsistent structured and unstructured data Improving data quality … Care about data quality, clean engineering and genuine ownership Have experience building SaaS products or other production platforms Experience with Azure, Node.js, data warehousing, Delta Lake or vector databases would be valuable. You don’t need previous private-markets experience. What matters is strong engineering judgement ...

build and optimise data pipelines across Pricing, Underwriting and Claims. You’ll work on a modern Azure-based data platform, consolidating capabilities on Databricks, Delta Lake and Unity Catalog to support analytics and AI initiatives. You’ll collaborate with Data Architects and Analysts, delivering scalable, secure data solutions ...

Platform Engineering team, you'll own data pipelines from ingestion through transformation, governance and consumption. You'll work with technologies including Databricks, Spark and Delta Lake to make complex data reliable, secure and ready for use across OQC's customer-facing applications. You’ll collaborate across Software, Platform … across engineering teams to deliver reliable data foundations for customer-facing products. What We're Looking For Strong hands-on experience with Databricks, Spark, Delta Lake and SQL. Experience building and operating streaming and batch data pipelines. Strong understanding of data quality, lineage, access control and dataset governance. ...

growth. Skills, Knowledge & Expertise Experience and knowledge: Experience of designing and implementing an Azure analytical platform including Azure Databricks, Azure Data Factory, Azure Data Lake Storage, Delta Lake, Unity Catalog, PySpark, SQL, Python, and Azure DevOps. Experience of working with senior stakeholders to articulate the strategy, objectives ...

Medallion Architecture, and ensuring our Databricks platform is secure, governed, and production ready. Key Responsibilities Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake. Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms). Implement strong … clusters. Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets. Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required. Partner with analysts and data scientists to ensure that high-quality, well‐modelled ...

Medallion Architecture, and ensuring our Databricks platform is secure, governed, and production ready. Key Responsibilities: Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake. Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms). Implement strong … clusters. Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets. Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required. Partner with analysts and data scientists to ensure that high-quality, well-modelled ...

Must Have: At least 7 years as an Engineer in a Data Field Strong Azure data skills, including: Azure Data Factory V2 Azure Data Lake Storage V2 Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell … Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software development practices, such as SOLID principals, structuring code, testing, IOC, etc. Strong repositories, CI/CD skills. Strong knowledge of Kimball ...

extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background in modern software development practices ...

Skills Strong experience designing and delivering modern data platforms using Microsoft Fabric, Azure Databricks, Azure Data Factory, Azure Synapse Analytics, Azure SQL and Data Lake technologies . Proven experience implementing Lakehouse, Data Warehouse and Medallion Architecture patterns, including data modelling, governance and performance optimization . Expertise in building scalable … Python, Spark and PySpark for data engineering, analytics and automation workloads. Experience with Azure Databricks and/or Microsoft Fabric engineering workloads, including notebooks, Delta Lake, data transformation, optimization and operationalization of data solutions. An understanding of data governance, security, lineage and data quality, using tools and frameworks ...

systems design within a modern cloud‐based environment Advanced experience working and understanding the tradeoffs of at least one of the following Data Lake table/file formats: Delta Lake, Parquet, Iceberg, Hudi Experience working with lakehouse/medallion architectures in Databricks Experience with experimentation support tooling ...

scalable, maintainable, and aligned with organisational standards and governance requirements. Design, build, and optimise data pipelines on Databricks using PySpark and Spark SQL, applying Delta Lake and a medallion/Lakehouse architecture. Implement data validation and reconciliation checks, and automated testing across pipelines to ensure trusted, accurate outputs. ...

make sound engineering decisions. Strong communication skills, with the ability to explain complex technical concepts clearly to both technical and non-technical audiences. Databricks: Delta Lake, Unity Catalog, Workflows, Asset Bundles, Notebooks, Compute Management, Delta Live Tables, Spark Declarative Pipelines and platform optimisation. Microsoft Fabric: Lakehouse, Data ...

customer in Central London. Key Responsibilities Data Lakehouse Architecture: Design and optimize a high-performance lakehouse pattern utilizing Microsoft Fabric OneLake and Azure Data Lake Storage Gen2 (ADLS Gen2). Data Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers … OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls ...

managing complex datasets within a recognised database environment (preferably PostgreSQL/PostGIS) and/or a cloud storage facility (such as a Lakehouse, Delta Lake or Data Lake solution) Excellent organisational skills and the ability to manage competing priorities Experience of using GIS systems, such as ArcGIS ...

large scaleStrong programming skills in Python, with applied experience in data engineering workflowsExpertise in PySpark for distributed data processingHands-on experience with Databricks, including Delta Lake and performance optimizationKnowledge of Azure Data Factory for orchestration of pipelines (ETL/ELT)Familiarity with AI/ML pipeline development, including ...

data platforms. Expertise with one or more of the following technologies: AWS (Redshift, Glue, S3, Bedrock) Microsoft Azure (Synapse, Data Factory, Azure SQL) Databricks (Delta Lake, ETL pipelines, ML workloads) Snowflake Strong data modelling experience across enterprise-scale environments. Experience building modern AI-enabled data architectures. Knowledge ...