1 to 25 of 41 Delta Lake Jobs in the UK

their own workflows. We are in the process of a major technology migration, moving our core data pipelines from generating HDF5 files into generating Delta Tables. This will improve efficiency, ensure future scalability and modernise the architecture. We are looking for an experienced lead to head up a small … cutting‐edge FinTech company with the potential to grow your technical, finance and cloud‐ & data‐engineering skills. Responsibilities Technical leadership and management of the delta lake migration team, organising work into 2‐week sprints, upwards reporting on progress and blockers. Development, testing and migration of some ...

+ Bonus Our client is in the process of a major technology migration, moving their core data pipelines from generating HDF5 files into generating Delta Tables. This will improve efficiency, ensure future scalability and modernise the architecture. They are looking for an experienced lead to head up a small … cutting-edge FinTech company with the potential to grow your technical, finance and cloud- & data-engineering skills. Responsibilities Technical leadership and management of the delta lake migration team, organising work into 2-week sprints, upwards reporting on progress and blockers. Development, testing and migration of some ...

into the Bronze layer, working in partnership with the Development and Application Management team. Design and optimise ETL/ELT pipeline frameworks using Databricks, Delta Lake and Azure Data Factory. Ensure Silver and Gold layer data products are fit for purpose for analytics, pricing, AI and ML model … scalability. Experience in regulated financial services, ensuring compliance with industry standards. Deep expertise in data modelling — conceptual, logical and physical. Data warehousing and data lake architecture for high‐performance analytics. ETL/ELT pipeline development and optimisation to support large‐scale data processing. Data integration across structured and unstructured ...

into the Bronze layer, working in partnership with the Development and Application Management team. Design and optimise ETL/ELT pipeline frameworks using Databricks, Delta Lake and Azure Data Factory. Ensure Silver and Gold layer data products are fit for purpose for analytics, pricing, AI and ML model … scalability. Experience in regulated financial services, ensuring compliance with industry standards. Deep expertise in data modelling — conceptual, logical and physical. Data warehousing and data lake architecture for high‐performance analytics. ETL/ELT pipeline development and optimisation to support large‐scale data processing. Data integration across structured and unstructured ...

analytics tasks and queries. Exposure to Java script technologies (preferably React) Exposure to Azure Databricks environment is desirable (PySpark, notebooks, integrate repos, job workflows, delta lake, unity catalog etc) Exposure to azure environment and various interconnected azure components (Functions, App Service, Containers etc.). Knowledge on Financial domain … Bash, PySpark Web frameworks: Django, Flask, or FastAPI. Frontend Technologies: ReactJS, Java Script Azure : Azure Databricks, Azure Synapse Analytics, Azure Functions, Azure Data Lake Storage Gen2, Azure Event Grid, Azure Event Hubs, Azure Service Bus, Azure Key Vault, Azure Monitor, Azure Log Analytics, Azure API Management (APIM), Azure DevOps. ...

full data engineering lifecycle. Key Responsibilities • Design and develop scalable ETL/ELT pipelines using Databricks, Python, SQL, and Spark • Build and optimise Delta Lake architectures and data pipelines on AWS • Implement ingestion frameworks using Databricks Autoloader • Develop transformation workflows using Lakeflow/Delta Live Tables • Work … best practices Essential Skills & Experience • Strong commercial experience as a Senior Data Engineer • Proven expertise with Databricks on AWS (essential) • Strong understanding of Spark, Delta Lake, and Lakehouse architecture • Hands-on experience with Databricks Autoloader and Lakeflow/DLT • Experience using Data Asset Bundles within Databricks environments • Strong ...

dimensional modelling) for consistent implementation across data solutionsEvaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patternsDesign technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)Create and maintain architecture documentation, including data flow diagrams, architecture … recommend new technologies and tools that enhance Arriva's data and BI capabilitiesStay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutionsConduct proofs-of-concept and technical assessments to validate new technologies before adoptionMonitor ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

modelling) for consistent implementation across data solutions* Evaluate and recommend architectural approaches for data storage, processing, and consumption, including lakehouse, data warehousing, and data lake patterns* Design technical architecture for cloud-based data platforms (AWS preferred: Redshift, Athena, S3, Glue)* Create and maintain architecture documentation, including data flow diagrams … recommend new technologies and tools that enhance Arriva's data and BI capabilities* Stay current with emerging trends including open table formats (Apache Iceberg, Delta Lake), data observability, real-time/streaming architectures, and cloud-native solutions* Conduct proofs-of-concept and technical assessments to validate new technologies ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

Microsoft Azure and Databricks Key Responsibilities: Design and implement scalable, secure cloud-based data architecture (Azure & Databricks) Develop optimised data models and pipelines using Delta Lake and Azure services Define data standards, policies, and governance practices aligned with compliance Enable real-time analytics and machine learning use cases … internal stakeholders and third-party vendors Key Skills & Experience: Proven background designing and delivering enterprise-scale data platforms Strong knowledge of Microsoft Azure, Databricks, Delta Lake, and data warehousing Advanced data modelling and ETL/ELT optimisation experience Familiarity with regulatory frameworks such as IFRS 17 , BCBS ...

framework for RC, PFE and EAD calculations, including netting sets, supervisory factors and collateral/CSA logic.* **Design a modern Databricks architecture** using Delta Lake, Unity Catalog and Spark/PySpark to support versioned, reproducible and regulator‐defensible exposure calculations.* **Develop high‐performance compute pipelines** that support intraday … Deep understanding of SA‐CCR** (RC, PFE, EAD), exposure methodology, netting sets, supervisory parameters and collateral/CSA design.* **Strong Databricks engineering capability**, including Delta Lake versioning, Unity Catalog governance, Spark/PySpark optimisation and secure data architecture.* **Experience designing governed, audit‐ready systems**, with a focus ...

Doing Designing and building end-to-end data pipelines using Databricks on Azure or AWS Developing modern lakehouse architectures using: Databricks (Spark, SQL, Delta Lake) Azure Data Lake Storage or Amazon S3 Writing high-quality PySpark, Python, and SQL for ingestion, transformation, and modelling Implementing Bronze/ ...

client-facing documentation. Preferred Criteria Experience with real-time data processing (Kafka, Kinesis, Azure Event Hub). Knowledge of big data storage solutions (Delta Lake, Parquet, Avro). Experience with data visualization tools (Power BI, Tableau, Looker). Understanding of AI/ML concepts and collaboration with ...

DevOps, Power BI, Fabric workspaces AWS cloud, S3, cross-cloud connectivity, API Gateways GCP (BigQuery, GCS, Dataflow) Deep expertise in: Databricks (Lakehouse architecture, Delta Lake, Unity Catalog, Delta Sharing, Pyspark, Spark SQL etc.) Snowflake (data warehousing & data sharing, Iceberg, ) Strong understanding of: Data architecture patterns, industry benchmarking ...

engineering, RAG architectures, vector databases, and 2) **Data Platforms****:** AWS Glue, SageMaker Unified Studio, Databricks on AWS, Snowflake on AWS, Redshift, EMR, Apache Iceberg, Delta Lake.• Position AWS solutions effectively against other cloud platforms' offerings in both generative AI (Azure OpenAI, Vertex AI) and data platforms (Azure Synapse, BigQuery … Proficiency in Python, SQL, and Spark with hands-on experience in:* Generative AI: LangChain, vector databases, embedding models.* Data Engineering: PySpark, Apache Iceberg/Delta Lake, orchestration tools.* A proven ability to articulate both visionary AI possibilities and practical data platform requirements to diverse audiences.#LI-LP2**About Rackspace ...

reliability. Collaborate with analysts and AI engineers to deliver robust and reusable data products. Manage data lakes and warehouses using formats such as Delta Lake and Parquet. Implement best practices for data governance, performance, and security. Continuously evaluate and adopt new technologies to evolve Movar's data platform. ...

modern data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering practices (Airflow, dbt, Prefect) and governance (Collibra, Purview) Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP Lead MLOps strategy (MLflow, Kubeflow, SageMaker, Vertex ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

connectivity issues across Foundry, Databricks and AWS. Desirable Skills Experience with Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder or Code Repositories. Experience with Delta Lake, Parquet, Iceberg, CSV and JSON. Data governance, lineage, secure data-sharing and access approval experience. Python, PySpark or SQL development experience. CI/ ...

data engineering experience — proven track record building big data processing platforms in production, moving beyond scripting to robust engineering solutions (e.g., Databricks/Delta Lake, Snowflake, BigQuery). Hands-on experience architecting Data Warehouses and Data Lakes. API and service design maturity — experience designing multi-service systems with ...

cloud-native services across AWS and Azure Develop scalable ETL/ELT frameworks and optimise performance across distributed compute Implement Lakehouse best practice including Delta Lake, medallion architecture, and governance patterns Collaborate with senior stakeholders to translate business requirements into technical delivery Ensure platform reliability through monitoring, optimisation …/CD deployment standards Essential Skills & Experience Databricks engineering - strong hands-on experience Azure data stack - Data Lake, ADF, Functions, DevOps AWS data services - Glue, Lambda, S3, EMR (or equivalent) Python & SQL - production-grade coding CI/CD - Git-based workflows, automated deployments Stakeholder communication - able to operate autonomously ...

Role Design and build scalable ETL/ELT pipelines in Microsoft Fabric using PySpark and SQL Develop transformation logic within Fabric Lakehouses using Delta Lake and medallion architecture principles Integrate data from ecommerce, ticketing, and marketing platforms into a central data layer Connect sources such as GA4, Google ...

multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience ...