76 to 100 of 150 Delta Lake Jobs in the UK

PySpark and experience optimising large-scale distributed processingBig data experience — designing and operating systems for datasets and workloads well beyond single-machine scaleApache Iceberg, Delta Lake, or similar open table formats and experience with lakehouse architectures, Databricks, Snowflake, or comparable data platformsCI/CD tooling such as GitLab ...

divergent source systems into canonical models across multiple products. Strong grounding in Lakehouse architecture trade-offs: table formats, query engines, ingestion patterns (Iceberg vs Delta; Dremio vs Trino vs Snowflake; streaming vs batch). Hands-on experience with: Dremio, Snowflake, or Databricks AWS data services (S3, Glue, Athena, Lake … Stream processing (Apache Flink, Spark Structured Streaming, or equivalent) SQL and query optimisation at scale Data and table formats (Parquet, Avro, Apache Iceberg or Delta Lake) Experience designing SaaS data models, tenant isolation, shared schemas, access patterns, at scale. Ability to communicate architecture decisions to engineers, product leadership ...

secure handling of inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena ...

MLOps Pipelines: Architect and automate robust pipelines for data preparation, model training, evaluation, and automated deployment leveraging Databricks Workflows, MLflow, and Delta Lake. Model Serving & Performance Optimization: Deploy, scale, and optimize low-latency real-time endpoints and high-throughput batch pipelines using Databricks Model Serving, monitoring compute resource allocation … Projects), plus standard ML libraries (e.g., scikit-learn, XGBoost, LightGBM, PyTorch). Enterprise Governance - Proven hands-on experience structuring environments with Unity Catalog and Delta Lake (ACID transactions, time travel, Z-ordering, Liquid Clustering). DevOps & Cloud - Proficiency in automated deployments using Databricks Asset Bundles (DABs) or Terraform ...

with business stakeholders and understand the broader context behind technical requirements. Nice-to-Have Skills Experience with Apache Iceberg or similar table formats (e.g., Delta Lake, Hudi). Familiarity with CI/CD platforms such as GitLab CI, Jenkins, or GitHub Actions. Exposure to data quality frameworks such ...

writing and troubleshooting transformations, validation and reconciliation queries. Experience scheduling, monitoring and recovering pipelines using cloud‐native or equivalent orchestration tools. Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading. Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud ...

Demonstrable data modelling experience, including practical experience with medallion architecture, and comfortable contributing technical input into modelling discussions alongside IT Experience with Databricks, including Delta Lake; familiarity with Unity Catalog and semantic layer/consumption-layer tooling is a plus (Power BI & Databricks Dashboards) Power BI experience, able ...

hands-on experience with Azure Data Factory. Hands on knowledge of the Microsoft Fabric architecture. Strong hands-on experience with Azure Databricks, PySpark, and Delta Lake. Proven experience architecting complex ETL/ELT data integration solutions. Strong experience working with XML, XSD schema validation, XPath, and XSLT transformations. Experience ...

secure handling of inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open‐source frameworks Familiarity with AWS‐native data services such as Glue, Lake Formation, Redshift, or Athena ...

modern data platforms or lakehouses. Modern lakehouse mindset: Deep understanding of open lakehouse design and open table formats (such as Apache Iceberg or Delta Lake), keeping data open and portable. Practical, multi-paradigm data modelling: Deep, hands-on experience designing data models across Data Vault 2.0, dimensional modelling ...

writing and troubleshooting transformations, validation and reconciliation queries. Experience scheduling, monitoring and recovering pipelines using cloud-native or equivalent orchestration tools. Experience with Delta Lake, including transactional tables, schema changes and repeatable batch loading. Experience using Terraform or equivalent infrastructure as code to provision and maintain consistent cloud ...

PySpark. You write reusable, parameterized functions and work comfortably with JSON, CSV, and Parquet. Platform knowledge Deep experience with Databricks (Declarative Pipelines, DABS, Delta Lake, Spark optimization, job orchestration). Familiarity with the Azure ecosystem. Working understanding of Unity Catalog. About you Around 6 or more years ...

device data services (desirable) Experience with containerisation, Kubernetes, AKS, API management, application gateways, or equivalent cloud application hosting technologies (desirable) Experience with Databricks, Spark, Delta Lake, Unity Catalog, Azure Data Factory, SAP BW/4HANA, S/4HANA, OData, CDS views, CDC patterns, or equivalent enterprise data technologies ...

warehouse platforms (e.g., SAP BW, Progress, Snowflake, Synapse, or similar).Hands-on experience building and delivering solutions in Databricks, including working knowledge of Delta Lake, Databricks Workflows, Genie spaces, and Unity Catalog controls. Strong command of SQL, Spark, Python, and modern ELT/orchestration tooling, with an automation ...

engagement platforms, event tracking, identity resolution, and customer data models. Experience building and optimising batch and streaming data pipelines using SQL, Python, PySpark, and Delta Lake. Familiarity with dbt for data transformations and orchestration tools such as Airflow, Dagster, or Prefect. Experience designing and delivering secure, resilient API, webhook ...

search platform that handles diverse query types (keyword, semantic, or hybrid) across large structured or semi-structured datasets. Hands-on experience with data lake, warehouse or lakehouse architectures at scale: Delta Lake or Apache Iceberg, schema evolution patterns, partition pruning, and the trade-offs between query performance ...

reusable frameworks and accelerators. What You’ll Be Doing Building production-grade data pipelines with Databricks, PySpark and SQL Developing Lakehouse architectures using Delta Lake, Unity Catalog and medallion frameworks Gathering requirements and providing technical guidance to client teams Embedding CI/CD, automated testing and Infrastructure ...

Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data modelling and lakehouse design (bronze, silver, gold) Terraform, CI/CD and pipeline orchestration A track record of setting standards ...

Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data modelling and lakehouse design (bronze, silver, gold) Terraform, CI/CD and pipeline orchestration A track record of setting standards ...

with Azure Databricks, Spark SQL, Databricks SQL and PySpark. Strong understanding of Apache Spark execution, data distribution, joins, shuffles and performance optimisation. Experience with Delta Lake and modern lakehouse architecture. SQL/Synapse to Databricks Migration Demonstrable experience migrating SQL workloads from Synapse Dedicated SQL Pools or similar ...

including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. Deep lakehouse and data platform expertise(e.g., Delta Lake concepts, schema evolution, data quality frameworks, lineage/metadata, and governance patterns) applied in production environments. Strong DevSecOps and platform engineering capability, including ...

personNice to have, not requiredDo not rule yourself out if you do not have these. They are a bonus, not a checklist. Experience with Delta Lake, Iceberg, Spark or Databricks, PalantirExperience optimising LLM serving with tools such as vLLM, TGI or TensorRT LLMSearch and ranking experience. For example ...

cloud databases Proven track record with complex data migration projects (terabyte+ datasets, multiple legacy source systems, structures and unstructured data) Proficiency with Parquet/Delta Lake or other modern data storage formats Experience with streaming architectures using Kafka, Event Hubs, or Kinesis for real-time data processing Knowledge ...

career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable data products. Build robust … data engineering capability. Essential Skills & Experience Proven hands‐on experience building and supporting production‐grade data pipelines using Databricks, PySpark, Spark SQL and Delta Lake. Experience designing and implementing Lakehouse architectures, including Medallion (Bronze, Silver, Gold) data models. Expertise in developing scalable ETL/ELT pipelines capable of handling ...

help the team do better and be better, again and again Write maintainable, testable code and enjoy providing code reviews Be experienced with Databricks, Delta Lake and Lakehouse architecture for efficient data management; experience with ETL processes and optimizing data pipelines for performance Be strong in Python ...