1 to 25 of 41 Delta Lake Jobs in the UK

working on behalf of a major client in the energy sector. The position focuses on designing, developing, and maintaining Azure-based data platforms-particularly Delta Lake and modern analytics services-while providing architectural leadership and hands-on engineering expertise. The role operates on a hybrid model, requiring three … days per week onsite in Glasgow. Key Responsibilities Develop and maintain Azure data architecture , including Delta Lake, ensuring compliance with data governance and security standards. Analyse and resolve data integration and system interoperability challenges across complex environments. Design and implement Azure data solutions using services such as Databricks ...

with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … streaming platforms (e.g. Event Hubs, Kafka), and legacy databases into ADLS Gen2 and OneLake Develop structured data models aligned to Lakehouse, Medallion Architecture, and Delta Lake patterns Continuously profile, debug, and optimise Spark jobs, SQL queries, and Python scripts for maximum performance and cost-efficiency at scale Champion ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

work at home 100% of the time. Your Role: Design, develop, and optimise large-scale cloud data platforms using Azure Data Factory, Data Lake, Key Vault, Azure Functions, Databricks, Delta Lake, PySpark, and Unity Catalog. Collaborate with underwriting, actuarial, Delegated Authority, bordereaux, exposure management, reinsurance, finance … continuous improvement. Your Skills: Expert-level hands-on experience with Azure Data Services and Databricks in enterprise, highly regulated environments. Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns, distributed compute, data modelling, and performance optimisation. Strong programming skills in Python, PySpark, and Spark SQL, with experience implementing ...

fully hands-on engineering role. Key responsibilities Design and build enterprise data platforms using Microsoft Azure services including Azure Data Factory, Azure Data Lake Storage, Azure Key Vault, Azure Functions and Azure Databricks Develop scalable pipelines using Delta Lake, PySpark and Unity Catalog Work with underwriting, actuarial … platform Required experience and skills Strong engineering experience with Microsoft Azure data services and Azure Databricks in large scale environments Deep knowledge of Delta Lake, medallion architecture, distributed compute and lakehouse data platforms Strong development skills in Python, PySpark and Spark SQL Experience implementing CI/ ...

modern cloud data platform. Key Responsibilities Design and implement scalable data models within Databricks. Transform raw datasets into curated, analytics-ready data products using Delta Lake and Spark. Build and maintain semantic layers for business reporting. Translate stakeholder requirements into technical specifications. Define KPIs and controlled vocabulary across … governance standards. Support CI/CD processes and version control workflows. Required Experience Strong SQL expertise (data modelling & optimisation). Hands-on Databricks experience (Delta Lake + Spark). Experience with dbt or similar transformation frameworks. Python for data manipulation and automation. Azure cloud experience. Familiarity with ...

practices for platform components including Lakehouse architecture, MLOps, and data governance. Platform Engineering Architect and guide implementation of Databricks workspace setup, clusters, Unity Catalog, Delta Lake, workflows, and model serving. Lead the integration of the platform with key internal systems such as Radar, EDP (Enterprise Data Platform … risk assessments. Required Skills & Experience: 8-12+ years of experience in data/AI engineering or cloud architecture roles. Strong expertise with Databricks (Delta Lake, Unity Catalog, MLflow, Workflows). Deep understanding of AWS cloud architecture and services relevant to data/AI workloads. Hands-on experience ...

energy market data using Python, SQL and Spark. Support data flows ensuring generation and storage assets deliver the correct contracted schedules. Design and manage Delta Lake tables and SQL databases, applying strong data modelling and optimisation strategies. Develop monitoring dashboards and alerting to ensure transparency around pipeline health … similar production-focused role Strong Python (Pandas) and SQL skills Experience building and maintaining data pipelines Hands-on experience with Spark, Databricks or Delta Lake/Lakehouse architectures Familiarity with cloud data platforms, particularly within Azure environments Good understanding of data modelling, storage formats and performance optimisation Experience ...

DSFS data model and existing Netezza structures. Produce the EDW logical and physical data models, dimensional models for the Semantic Layer, and Data Lake zone architecture (Bronze/Silver/Gold). Define Enterprise Semantic Layer design, including pre-computed aggregations, dimensional models, and Tableau connectivity specifications. Lead source … data architecture, with at least 3 years leading enterprise data model design. Experience designing Lakehouse architecture: medallion architecture (Bronze/Silver/Gold), Delta Lake, or equivalent. Proficiency in dimensional modelling (Kimball methodology) and enterprise data modelling. Experience with Databricks (Unity Catalog, Delta Lake) or Snowflake ...

PySpark and Spark SQL Proficiency with Python for data engineering Experience delivering solutions on cloud data platforms ( AWS preferred , Azure acceptable) Understanding of data lake and lakehouse architectures Ability to work autonomously and collaboratively within delivery squads Experience working within Central Government departments (highly desirable) Desirable Azure Data Lake, Synapse, Delta Lake Streaming technologies (Kafka, Event Hubs ...

Strong analytical mindset with the ability to drive business impact through data Technical Environment Languages: Python, SQL, Bash Cloud: Azure, AWS Data & Platforms: Snowflake, Delta Lake, Redis, Azure Data Lake Tools: Airflow, DBT, Docker Infrastructure & Operations: Terraform, GitHub Actions, Azure DevOps, Azure Monitor Desirable Skills Experience with … data platforms such as Snowflake or Azure Data Lake Experience deploying models as APIs (e.g. FastAPI, Azure Functions) Knowledge of monitoring, model performance tracking and observability practices Familiarity with orchestration tools such as Airflow or Azure Data Factory ...

with excellent communication skills. Technical Stack: Languages: Python (primary), SQL, Bash Cloud: Azure, AWS Tools: MLflow, Azure ML, Airflow, Docker, Fargate, FastAPI Data: Snowflake, Delta Lake, Redis, Azure Data Lake Infra & Ops: Terraform, GitHub Actions, Azure DevOps, Grafana, Azure Monitor Desirable skills/knowledge/experience: Knowledge … data platforms (e.g. Snowflake, Azure Data Lake). Experience deploying models as APIs (FastAPI, Azure Functions). Understanding of monitoring, model performance tracking, and observability best practices. ...

Responsibilities Lead the design, development, and deployment of scalable, secure, and cost-effective distributed data solutions using Azure services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory). Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability … frameworks, templates, and patterns to accelerate delivery and ensure consistency across projects. Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven ...

Responsibilities Lead the design, development, and deployment of scalable, secure, and cost-effective distributed data solutions using Azure services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory). Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability … frameworks, templates, and patterns to accelerate delivery and ensure consistency across projects. Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven ...

help modernise legacy data estates and embed strong data engineering standards. Key Responsibilities: Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance ...

/ELT pipelines and scalable data architectures . Hands-on experience with Azure cloud technologies , including: Azure Databricks Azure Data Factory Azure Data Lake Experience working with Delta Lake architecture and modern data warehousing patterns . Experience with Databricks components , including: Delta Live Tables Auto Loader ...

transformation Hybrid working with flexibility and autonomy Key Experience Deep end‐to‐end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python … Scala or SQL for data engineering Experience building scalable ETL pipelines (Workflows, Delta Live Tables) Hands‐on experience with Azure, AWS or GCP (storage, compute, networking) Knowledge of Databricks security: RBAC, IAM, encryption Excellent analytical skills and the confidence to challenge thinking at all levels Bonus: Databricks certifications (Data ...

Lead the design, development, and deployment of scalable, secure, and cost-effective distributed data solutions [MA2.1]using Azure services (e.g., Azure Databricks, Azure Data Lake Storage, Azure Data Factory).• Architect and implement advanced data pipelines using Databricks, Delta Lake, Python and Spark, ensuring performance, reliability … level engineers, fostering a culture of continuous learning, innovation, and technical excellence. Experience needed: • Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory.• Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing.• Proven ...

engineering with Apache Airflow, dbt, and Prefect, ensuring performance, governance, and alignment with enterprise metadata standards (Collibra, Alation, Microsoft Purview). Drive adoption of Delta Lake, Iceberg, and Hudi to support scalable data lakehouse architectures. Ensure high‐quality, compliant, and reliable data foundations for ML and analytics workloads. ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...

Kimball, Data Vault). Experience exposing Lakehouse architectures to tools such as Power BI and Tableau. Hands-on experience in Databricks engineering: Spark, Delta Lake, PySpark. Skilled in building scalable ETL pipelines using Workflows and Delta Live Tables. Proficiency in Python, Scala, or SQL. Knowledge of Databricks ...