1 to 25 of 35 Delta Lake Jobs in the UK

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

solutions. Key Responsibilities Design, develop, and maintain scalable data pipelines using Python and Databricks Build and optimise ETL/ELT workflows leveraging Databricks (Spark, Delta Lake) Develop robust data models and data architectures on Lakehouse platforms Implement and manage data workflows within the Databricks ecosystem Ensure high data … drive engineering best practices Skills & Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms ...

pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED Please either ...

ingest data from legacy systems and third-party sources. Develop and maintain the Bronze, Silver, and Gold layers of the lakehouse architecture using OneLake, Delta Lake, and Apache Spark within Fabric. Implement data transformation logic using PySpark, SQL, and Fabric Notebooks; ensure data quality, lineage, and cataloguing … Azure Data Factory (ADF) for orchestration and data movement Strong proficiency in PySpark, SQL, and Python for large-scale data transformation Strong experience with Delta Lake, Apache Spark, and OneLake architecture Good knowledge of Microsoft Purview for data governance, cataloguing, and lineage Good experience with Azure DevOps ...

scalability, security, and governance. Act as Databricks champion: define standards, reference architectures, accelerators, and reusable assets. Lead architectural design for Databricks (workspaces, clusters, jobs, Delta Lake, Unity Catalogue, MLflow) and integrations with cloud services. Partner with pre-sales teams to shape propositions, run discovery, produce high-quality proposals … encryption, auditing, lineage). Essential Requirements Significant Databricks experience required, including production-scale deployments and optimisation. Strong hands-on capability with Spark/SQL, Delta Lake, orchestration, and data engineering patterns. Extensive experience designing cloud data platforms (AWS, Azure, or GCP) and integrating with enterprise systems. Proven stakeholder ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

understanding of cloud security, access control, and governance principles Strong software engineering fundamentals and automation mindset Nice-to-have: Enterprise-scale data platform experience Delta Lake, Delta Live Tables (DLT), and Databricks SQL Azure monitoring and observability tooling Event-driven or streaming architectures Experience working across platform ...

Microsoft Azure and Databricks Key Responsibilities: Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience: Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable 'single version of the truth' for reporting and analytics; - Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS); - Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … . Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing). - Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach. - Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

modern cloud data engineering. Hands-on experience with Microsoft Fabric, including Lakehouse, Warehouse, Pipelines, Dataflows, Notebooks and Power BI. Good understanding of data lake, lakehouse and warehouse architecture. Experience building data ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. … PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables and common enterprise data formats. Knowledge of data governance, access control, data quality, metadata and lineage. Good documentation skills and the ability to produce ...

engineering processes. Mentor junior engineers and support knowledge-sharing across teams. Key Responsibilities: Design, build, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake. Develop efficient ETL/ELT workflows to process large volumes of structured and unstructured data. Implement data governance, security, and compliance standards. Work … pipeline deployment and monitoring. What We Are Looking For: 5+ years of experience in data engineering or related roles. Strong expertise in Databricks, Spark, Delta Lake, and cloud data platforms (AWS, Azure, or GCP). Proficiency in Python and SQL for data manipulation and transformation. Experience with ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure SQL, Key Vault) as they relate ...

opportunities 5. Technology Leadership (Azure + Databricks + Power BI) Lead architecture and execution of: Azure Data Platform (ADF, Synapse, Fabric, ADLS) Databricks (Lakehouse, Delta, ML workflows, PySpark pipelines) Power BI (semantic models, enterprise dashboards, self-service BI) Drive adoption of: Metadata-driven architectures Automation, orchestration, and reusable frameworks … code/data modernisation approach es Technical Expert iseStrong hands-on/architectural expertise in:Azure data ecosystem (ADF, Synapse, Fabric, AD LS)Databricks (Delta Lake, Spark, ML workflo ws)Power BI (enterprise analytics & semantic lay er)Strong grounding in:Data modelling (dimensional, domain-driv en)Data governance ...

with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation ...

reliability within production data environments. Experience working with cloud data services, ideally Azure, although AWS or GCP experience would also be relevant. Exposure to Delta Lake, Databricks Workflows, scheduling tools or lakehouse environments would be beneficial. An understanding of Power BI, reporting data models, governance, cataloguing or lineage ...

stakeholder engagement and collaboration. Design, build and maintain scalable data pipelines. Develop using Azure Databricks (PySpark, Spark SQL, Python). Contribute to Lakehouse architecture (Delta Lake patterns, Databricks capabilities). Build and manage workflows (Databricks Workflows, Delta Live Tables). Implement data governance (Unity Catalog – access, discovery … Strong Python, SQL, PySpark and Spark SQL capability. Experience with Lakehouse/modern data platform design. Hands-on with Databricks tools (Unity Catalog, Workflows, Delta Live Tables, Databricks SQL). Data ingestion from APIs and external sources. Geospatial data processing experience (real-world use cases). Experience with geospatial ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

building scalable data pipelines and data processing frameworks. Strong understanding of modern Lakehouse architecture principles. Experience working with: Unity Catalog/Databricks Workflows/Delta Live Tables/Databricks SQL. Experience integrating data from APIs and external data sources. Strong understanding of data modelling, data quality and performance optimisation … maintain high-performance data pipelines using Azure Databricks, PySpark, Spark SQL and Python. Contribute to the design and implementation of Lakehouse architecture using Delta Lake and Databricks best practices. Develop and manage data workflows using Databricks Workflows and Delta Live Tables. Utilise Unity Catalog to support effective ...

Looking For Proven experience delivering large-scale Databricks transformation projects Strong Financial Services or enterprise consulting background Deep expertise across Azure, Spark, Delta Lake and modern data engineering practices Experience building and leading high-performing technical teams Strong commercial awareness and stakeholder management experience Ability to lead complex ...

Commercial teams to deliver scalable solutions What you bring Must-haves 8+ years in data engineering/data architecture Deep experience with Databricks (Lakehouse, Delta Lake, Unity Catalog) Expert SQL and distributed computing knowledge Strong experience with data modelling (star/snowflake schemas) Proven experience migrating legacy data ...

least 3 years hands-on with Azure Databricks in production environments Proven experience designing and implementing medallion architecture (Bronze/Silver/Gold) using Delta Lake on Azure Databricks Strong hands-on experience with Unity Catalog — metastore setup, catalog and schema design, access controls, and governance policies Experience … configuring Azure Data Lake Storage Gen2 — containers, access control lists, lifecycle policies, and integration with Databricks Hands-on experience with Microsoft Fabric — specifically Mirrored Azure Databricks Unity Catalog and Power BI connectivity to Databricks SQL Warehouses Experience with enterprise data source integration — MS SQL Server (on-premises and Azure ...

multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience ...