1 to 25 of 26 Remote/Hybrid Delta Lake Jobs in the UK

OneLake and Databricks Unity Catalog. Build optimised Lakehouse and warehouse structures using Databricks SQL and Fabric SQL endpoints. Establish data pipelines using Databricks Workflows, Delta Live Tables (DLT), and Fabric Data Factory. Formulate cross‐platform data strategies utilising shared Delta Lake formats to eliminate data duplication. Implement … scalable lakehouse and warehouse solutions (Databricks SQL, Fabric SQL). Strong data pipeline development using Databricks Workflows, DLT, and Fabric Data Factory. Knowledge of Delta Lake and cross‐platform data strategies to reduce duplication. Data governance expertise, including Unity Catalog, RLS, and data lineage. Experience managing compute across ...

pipelines within a Databricks Lakehouse environment. You'll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering teams, you will help … Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding of Delta Lake, Delta Live Tables, and Medallion Architecture Proficiency in Python and SQL for data engineering and analytics TO BE CONSIDERED Please either ...

their own workflows. We are in the process of a major technology migration, moving our core data pipelines from generating HDF5 files into generating Delta Tables. This will improve efficiency, ensure future scalability and modernise the architecture. We are looking for an experienced lead to head up a small … cutting‐edge FinTech company with the potential to grow your technical, finance and cloud‐ & data‐engineering skills. Responsibilities Technical leadership and management of the delta lake migration team, organising work into 2‐week sprints, upwards reporting on progress and blockers. Development, testing and migration of some ...

into the Bronze layer, working in partnership with the Development and Application Management team. Design and optimise ETL/ELT pipeline frameworks using Databricks, Delta Lake and Azure Data Factory. Ensure Silver and Gold layer data products are fit for purpose for analytics, pricing, AI and ML model … scalability. Experience in regulated financial services, ensuring compliance with industry standards. Deep expertise in data modelling — conceptual, logical and physical. Data warehousing and data lake architecture for high‐performance analytics. ETL/ELT pipeline development and optimisation to support large‐scale data processing. Data integration across structured and unstructured ...

into the Bronze layer, working in partnership with the Development and Application Management team. Design and optimise ETL/ELT pipeline frameworks using Databricks, Delta Lake and Azure Data Factory. Ensure Silver and Gold layer data products are fit for purpose for analytics, pricing, AI and ML model … scalability. Experience in regulated financial services, ensuring compliance with industry standards. Deep expertise in data modelling — conceptual, logical and physical. Data warehousing and data lake architecture for high‐performance analytics. ETL/ELT pipeline development and optimisation to support large‐scale data processing. Data integration across structured and unstructured ...

dimensional modelling) for consistent implementation across data solutionsEvaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patternsDesign technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)Create and maintain architecture documentation, including data flow diagrams, architecture … recommend new technologies and tools that enhance Arriva's data and BI capabilitiesStay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutionsConduct proofs-of-concept and technical assessments to validate new technologies before adoptionMonitor ...

Microsoft Azure and Databricks Key Responsibilities Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

Microsoft Azure and Databricks Key Responsibilities: Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience: Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

reliable 'single version of the truth' for reporting and analytics; - Design and deploy secure, scalable ELT or ETL pipelines using Databricks (Unity Catalog), Delta Lake, and Azure Data Lake Storage (ADLS); - Implement shift-left security principles within the data ingestion layer, including automated data masking, encryption … . Experience - Implemented and operated cloud-based data platforms, ideally on Azure (experience MUST include build, not just maintenance of existing). - Delivered a Delta Lakehouse, Medallion Architecture, or similar curated data layering approach. - Created end-to-end ELT/ETL pipelines using Databricks, Spark, Azure DataFactory, or equivalent ...

years of relevant experience in data engineering and Azure platform administration. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation, and scripting. Azure … Bicep) across multiple environments. Scripting: PowerShell and Bash for platform automation. Monitoring & observability: Azure Monitor, Log Analytics, KQL. Big data file formats: Parquet and Delta Lake. Cloud-native data modelling and ETL/ELT frameworks on Azure. Good to Have Azure certifications: AZ-104, AZ-400, DP-203. Azure ...

years of relevant experience in data engineering and Azure platform administration. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation, and scripting. Azure … Bicep) across multiple environments. Scripting: PowerShell and Bash for platform automation. Monitoring & observability: Azure Monitor, Log Analytics, KQL. Big data file formats: Parquet and Delta Lake. Cloud-native data modelling and ETL/ELT frameworks on Azure. Good to Have Azure certifications: AZ-104, AZ-400, DP-203. Azure ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands-on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices. Must Have Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows). Python/PySpark: strong programming skills for data processing, automation ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

driven analytics for traders, including seasonality, correlations, regression, forecasting, and scenario modelling. Build scalable, reusable data pipelines in Databricks using PySpark, Spark SQL, Delta Lake, and Unity Catalog. Perform statistical and econometric analysis on large market and fundamentals time‐series datasets. Work directly with traders and analysts … guardrails). Productionise solutions with testing, observability, versioning, documentation, and CI/CD. Must‐Have Skills: Hands‐on Databricks and Spark expertise (PySpark, SQL, Delta, Unity Catalog). Strong data engineering background: ingestion, modelling, orchestration, performance tuning. Solid statistics/econometrics for market time series. Experience building and integrating ...

multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...

Modelling, Data Warehousing. Strong background in incident resolution, requests, changes and problem-solving within SLAs. Hands-on experience with Spark, Data Architecture, SQL and Delta Lake. Cloud development experience (AWS, GCP or Azure). Working knowledge of Medallion Architecture. Demonstrated capability in implementing and supporting pipelines in demanding environments. ...

architectural decisions, mentor colleagues and work on impactful projects that directly support business growth and transformation. Requirements... ✔ Strong commercial experience with Databricks, Spark and Delta Lake ✔ Proven experience designing and building scalable ETL/ELT pipelines ✔ Strong Python and SQL skills ✔ Experience working with cloud platforms (AWS preferred ...

architectural decisions, mentor colleagues and work on impactful projects that directly support business growth and transformation. Requirements... ✔ Strong commercial experience with Databricks, Spark and Delta Lake ✔ Proven experience designing and building scalable ETL/ELT pipelines ✔ Strong Python and SQL skills ✔ Experience working with cloud platforms (AWS preferred ...

Catalog grants, schema mismatches and query failures. Technical Environment : Palantir Foundry, Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder, Code Repositories, AWS, Databricks, Terraform, IaC, Delta Lake, Parquet, Iceberg, CSV, JSON, Python, PySpark, SQL, CI/CD, GitLab, GitHub Actions. ...

BMLL Technologies Limited in Greater London is seeking an experienced Technical Lead to manage the migration to Delta Lake. This role combines hands-on development, primarily in C++ and Python, with team leadership in Agile sprints. You will guide a small team through the migration process, collaborating with architects ...

BMLL Technologies is seeking a Lead Developer to oversee a pivotal technology migration. The role involves hands-on development primarily in C++ and Python, along with managing a small team in Agile sprints. You will ...