1 to 25 of 36 Delta Lake Jobs in the UK

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and Lakehouse solutions … time. Your Role: Define end-to-end data architecture for Azure-based data platforms using Azure Databricks, Azure Data Factory, ADLS Gen2, Delta Lake, Azure Synapse, and Power BI. Design scalable and secure lakehouse architecture using bronze, silver, and gold data layers. Lead architecture and design for data ...

/ELT pipelines using PySpark, SQL and orchestration frameworks Automate and modernise legacy data processes with reusable patterns Design robust data models (Delta Lake, schema evolution, metadata, lineage) Ensure data security, governance, and compliance (access control, encryption, lifecycle) Provide technical leadership, mentoring and design reviews Collaborate with stakeholders … analysis, profiling and insight delivery Cloud & Platforms Azure: Data Factory, Synapse, Databricks, ADLS Gen2, Event Hub, Purview AWS: S3, Glue, Athena, Kinesis, Lambda, EMR, Lake Formation Databricks: Delta Lake, Unity Catalog, Structured Streaming, MLflow SQL Server stack (SSIS, SSRS) Engineering Practices CI/CD for data platforms ...

Databricks (Unity Catalog) Build scalable lakehouse and warehouse solutions using Databricks SQL and Fabric SQL endpoints Develop robust data pipelines with Databricks Workflows, Delta Live Tables (DLT), and Fabric Data Factory Define and implement cross-platform data strategies using Delta Lake to minimise duplication Establish data governance … Databricks (Unity Catalog) architecture Proven experience building modern lakehouse and data warehouse platforms Hands-on experience with: Databricks SQL & Fabric SQL Databricks Workflows & Delta Live Tables (DLT) Fabric Data Factory Deep knowledge of Delta Lake and cross-platform data strategies Solid experience in data governance, including lineage ...

real-time insights. Key Responsibilities Architecture & Solution Design Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures … leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata ...

considering: We are seeking an experienced Azure Data Engineering Lead with deep expertise in Azure Databricks, Azure Data Factory (ADF), PySpark, SQL, Delta Lake, and Azure cloud-native data services to drive the design and delivery of enterprise-scale data platforms for leading banking and financial services clients. … optimization, performance tuning, joins, window functions, and stored procedures. Extensive experience in developing and optimizing large-scale ETL/ELT pipelines. Strong knowledge of Delta Lake, Lakehouse, and Medallion Architecture. Experience integrating data from databases, APIs, files, cloud storage, and streaming sources. Expertise in data quality, validation, reconciliation ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable "single version of the truth" for reporting and analytics;- Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS);- Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … Connect).Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing).- Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach.- Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

management, coaching and building high-performing architecture teams. Excellent communication, stakeholder management and influencing You will have demonstrable skills in the following : Azure Databricks (Delta Lake) Azure Data Lake Storage Gen2 Azure Data Factory Power BI Microsoft Purview Unity Catalog GitLab CI/CD MLflow, MLOps ...

retry mechanisms. Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing. Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault. Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling … reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications. Experience with entity resolution, record matching platforms, and third-party enrichment services. Experience with Delta Lake and Change Data Feed (CDF) patterns. Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation. Experience ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

Prefect , ensuring data pipelines are performant, governed, and aligned with enterprise metadata standards ( Collibra , Alation , Microsoft Purview ). Drive the adoption of Delta Lake , Iceberg , and Hudi for scalable data lakehouse architectures. Ensure high-quality, compliant data foundations for machine learning and analytics workloads. Cloud, Infrastructure & MLOps Champion ...

production experience is not required. A collaborative mindset and willingness to learn and communicate openly within the team. Nice to have Exposure to data lake concepts or formats such as Delta Lake or Iceberg. Any experience with search, graph, or large-scale analytical databases. Interest ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps principles, Agile Delivery. Advantageous Skills: Databricks, Palantir Foundry, Foundry Pipeline Builder, Ontology concepts, Delta Lake, Secure Data Environment (SDE) experience, NHS/Healthcare data experience. Government/(url removed)/Public Sector domain experience strongly preferred. Must ...

Data Engineer in cloud-first environments. Strong commercial knowledge of AWS services (e.g. S3, Glue, Redshift). Advanced PySpark and Databricks experience (Delta Lake, Unity Catalog, Databricks Jobs etc). Proficient in SQL (T-SQL/SparkSQL) and Python for data transformation and scripting. Hands-on experience with ...

Data Engineer in cloud-first environments. Strong commercial knowledge of AWS services (e.g. S3, Glue, Redshift). Advanced PySpark and Databricks experience (Delta Lake, Unity Catalog, Databricks Jobs etc). Proficient in SQL (T-SQL/SparkSQL) and Python for data transformation and scripting. Hands-on experience with ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

also contribute to best practice implementation and continuous improvement within cross-functional engineering teams. What You’ll Do Design and develop robust pipelines using DeltaLake, Spark Structured Streaming, and Unity Catalog Build real-time event-driven solutions with tools such as Kafka and Azure Event Hubs Apply DevOps principles ...

Databricks and Apache Spark. Build and optimize data workflows for batch and real-time processing. Transform and process large datasets using PySpark, SQL, and Delta Lake. Implement data ingestion from multiple sources (APIs, streaming, databases). Your Skills: Python proficiency. AWS : Cloud formation , Service Catalog , IAM , S3 , Glue , Lambda ...

data fidelity. Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts ...

model lifecycle: data prep, experimentation, deployment, and monitoring. Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. Ensure data quality, observability, lineage, and documentation across data … with OpenAI APIs, LLM workflows, and prompt engineering. Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. Experience with Databricks, Spark, and Delta Lake. Strong SQL skills with experience working on large datasets. Experience with AWS, including S3 and Lambda. Familiarity with Redshift, Snowflake, or other cloud ...

model lifecycle: data prep, experimentation, deployment, and monitoring. \n\n \n Build and optimize feature pipelines and scoring jobs using Python, Databricks, Spark, Delta Lake, and AWS. \n\n \n Use AWS services (S3, Redshift, Lambda) for data automation, orchestration, and scalable processing. \n\n \n Ensure data … prompt engineering. \n\n \n Solid machine learning fundamentals, including supervised learning, NLP, and feature engineering. \n\n \n Experience with Databricks, Spark, and Delta Lake. \n\n \n Strong SQL skills with experience working on large datasets. \n\n \n Experience with AWS, including S3 and Lambda. ...

solutions Designing scalable data and AI pipelines Implementing MLOps and LLMOps best practices We're interested in people with experience in: Python Databricks (PySpark, Delta Lake, Unity Catalog, MLflow, Mosaic AI) LLMs, GenAI and AI Agents MLOps and production ML systems CI/CD and DevOps ...