1 to 25 of 234 Delta Lake Jobs

business. As part of this journey, we're consolidating our data capabilities onto a single Azure Databricks platform, leveraging technologies including Azure Data Factory, Delta Lake, Unity Catalog and Power BI to unlock new insights and drive innovation. Joining our Enterprise Data Team , you'll play … driven initiatives across the organisation. You’ll be responsible for building and maintaining data pipelines using Azure Databricks, Azure Data Factory and Delta Lake, supporting the modernisation of legacy platforms while ensuring data quality, reliability and performance. The role will also involve implementing data governance and security controls ...

business. As part of this journey, we're consolidating our data capabilities onto a single Azure Databricks platform, leveraging technologies including Azure Data Factory, Delta Lake, Unity Catalog and Power BI to unlock new insights and drive innovation. Joining our Enterprise Data Team , you'll play … driven initiatives across the organisation. You'll be responsible for building and maintaining data pipelines using Azure Databricks, Azure Data Factory and Delta Lake, supporting the modernisation of legacy platforms while ensuring data quality, reliability and performance. The role will also involve implementing data governance and security controls ...

optimize enterprise-scale data platforms using the Databricks Lakehouse Platform . The ideal candidate will have deep expertise in Databricks architecture, Apache Spark, Delta Lake, data engineering, cloud platforms, and enterprise data solutions. Key Responsibilities Design and architect Databricks Lakehouse solutions for large-scaleenterprise data platforms. Lead … deployment, and optimization. Define Databricks workspace architecture, clusterstrategies, security models, and operational standards. Design and developscalable data pipelinesusing Databricks, Apache Spark, PySpark, SQL, andDelta Lake . Implement data ingestionframeworks supporting batchand real-timedata processing. Design Delta Lake architectures including: Delta tables Medallion architecture (Bronze/Silver ...

solutions using the Databricks Data Intelligence Platform. The ideal candidate should have strong hands-on experience with Databricks, Lakehouse architecture, Apache Spark, Delta Lake, Unity Catalog, cloud platforms, data engineering, and modern data/AI architectures. The candidate will work closely with business stakeholders, data engineers, developers, cloud … across development, testing, and production environments. Define target-state architecture for data engineering, analytics, BI, machine learning, and AI workloads. Architect solutions using Databricks, Delta Lake, Apache Spark, Unity Catalog, Databricks SQL, Workflows, and Lakehouse Federation. Design and implement enterprise data governance, security, access control, lineage, and data ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery. As a key leader within our AI & Data … value pools, translate business needs into Databricks-led solution designs, build prototypes and proof points. • Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic ...

build, test and deploy scalable batch and streaming data pipelines, AI-ready data products, feature pipeline and analytics solutions on the Databricks Platform using Delta Lake, Spark SQL/PySpark, Unity Catalog and Databricks SQL.Deliver solutions across the full data & AI engineering lifecycle, from requirements analysis and solution … major cloud platform (Azure, AWS, or GCP).Strong experience designing and building scalable data platforms, data products and analytics solutions using modern lake house or cloud data warehouse technologiesProficiency writing advanced, highly optimised SQL and Spark (PySpark/Scala), with a focus on performance optimisation and maintainability.Designing and building ...

code reviews and engineering quality assurance, promoting clean, maintainable, high-quality code and adherence to software engineering best practices.Optimise platform performance across Spark workloads, Delta Lake, data pipelines, storage, and compute to maximise scalability, reliability, and cost efficiency.Design and deliver enterprise-grade Machine Learning, Generative AI and Agentic … experience as a solution or platform architect, defining reference architectures, integration patterns, and target-state designs across enterprise Databricks environments.Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, PySpark, Databricks SQL, Databricks Workflows, Lakeflow, Lakehouse architecture, MLflow, AI/BI Genie, Mosaic AI, and Agent Bricks.Proven experience ...

AI. This is a hybrid Data Engineer role based in Belfast, offering the opportunity to work with modern cloud technologies including Databricks, AWS Glue, Delta Lake and Spark, while contributing to a wider programme of data transformation and continuous improvement. Key requirements include 2+ years' Data Engineering experience … premise, API and streaming sources Develop data architectures aligned with business requirements Build and maintain data models for BI and analytics Work with Delta Lake and modern data warehousing technologies Maintain Databricks notebooks, jobs and workflows Optimise Spark jobs and data workflows for performance and scalability Monitor, troubleshoot ...

growing data platform team. You'll play a key role in designing, building and optimising production-grade data pipelines across Databricks, Spark and Delta Lake, while helping shape the engineering standards and architecture behind the wider data platform. What you'll be doing: Build and optimise scalable Databricks …/Spark data pipelines across batch and streaming workloads. Develop Lakehouse solutions using Delta Lake and medallion architecture. Implement data quality, monitoring, governance and security across the platform. Work closely with Data Scientists, Analysts and Product teams to deliver high-quality data products. Contribute to engineering standards, code ...

build of large-scale data platforms using Azure Databricks and modern cloud technologies Implement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principles Enhance performance, manage cloud costs and ensure platform reliability for structured streaming workloads Define data governance, security and quality … optimization, data skew mitigation and query tuning Strong programming skills in Python with knowledge of modern software design principles Practical experience with structured streaming, Delta Lake and Delta Live Tables Proven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

data engineering platform Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security … boards The candidate Experience and knowledge requirements: Required Experience in data engineering disciplines, with advanced, hands‐on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governance Extensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehouses ...

data engineering platform• Build and optimize sophisticated ETL/ELT data pipelines using PySpark, SQL, and Python, orchestrating complex data workflows through Databricks Workflows, Delta Live Tables, Azure Data Factory• Implement and maintain Delta Lake storage architectures and Unity Catalog governance frameworks, ensuring data quality, security … architectural review boardsThe candidateExperience and knowledge requirements:RequiredExperience in data engineering disciplines, with advanced, hands-on expertise in Databricks platform including PySpark framework, Delta Lake storage format, and Unity Catalog governanceExtensive experience in designing advanced data models, including medallion architectures, data lakes, and enterprise data warehousesExpert-level proficiency ...

Asset Bundles. Key Responsibilities Design, build, test, and maintain scalable batch and streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design and manage Delta Tables, including … workflows. Package and deploy Databricks resources across environments using Databricks Asset Bundles and GitLab-based CI/CD pipelines. Integrate Databricks with Azure Data Lake Storage and Azure Data Factory for secure, reliable data processing. Optimize Spark workloads, clusters, jobs, and storage patterns for performance, scalability, reliability, and cost ...

platforms such as Azure, AWS or GCP* Define lakehouse standards, data modelling approaches and governance frameworks* Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies* Establish security, access control, data quality and compliance standards* Optimise performance, scalability and cloud cost management* Lead … experience* Strong experience as a senior or lead data engineer building and scaling production data platforms* Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows* Python and SQL proficiency* Infrastructure as code experience using Terraform, Bicep or CloudFormation* Cloud data platform experience, ideally Azure ...

platforms such as Azure, AWS or GCP Define lakehouse standards, data modelling approaches and governance frameworks Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies Establish security, access control, data quality and compliance standards Optimise performance, scalability and cloud cost management Lead … experience Strong experience as a senior or lead data engineer building and scaling production data platforms Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows Python and SQL proficiency Infrastructure as code experience using Terraform, Bicep or CloudFormation Cloud data platform experience, ideally Azure ...

platforms such as Azure, AWS or GCP * Define lakehouse standards, data modelling approaches and governance frameworks * Lead design of batch and streaming platforms using Delta Lake, Spark and related technologies * Establish security, access control, data quality and compliance standards * Optimise performance, scalability and cloud cost management * Lead … experience * Strong experience as a senior or lead data engineer building and scaling production data platforms * Deep hands on expertise in Databricks including Spark, Delta Lake and Databricks Workflows * Python and SQL proficiency * Infrastructure as code experience using Terraform, Bicep or CloudFormation * Cloud data platform experience, ideally Azure ...

GitHub Actions and Terraform setup for deployments Implement, run and monitor dbt tests to ensure data quality and consistent dimensional models Extend and maintain Delta Lake fact and dimension tables for analytics use cases, following a bronze/silver/gold-style data architecture Work closely with analysts … they bring clear value, including integrating AI tools for analysis and AI coding agents into day-to-day workflows Tech stack Azure Databricks (Spark, Delta Lake) SQL Server, Confluent Kafka Scala, SQL, dbt Git, GitHub Actions, Terraform Sigma (BI) Must-have experience Commercial experience as a Data Engineer ...

Data Engineer, you’ll focus on building and maintaining robust data pipelines within the AWS ecosystem, with a primary focus on the Delta Lakehouse Platform on Databricks.Responsibilities:Data Ingestion: Build and optimize data ingestion pipelines using AWS services such as Lambda, Kinesis, and Glue.Data Transformation: Develop, maintain, and optimize … transformation pipelines within our Delta Lake on Databricks.Data Modelling: Design and implement efficient and reliable data models to store and retrieve data.Collaboration: Work closely with data scientists, analysts, and stakeholders to understand data requirements and deliver optimal solutions.Data Quality: Ensure the reliability, integrity, and timeliness of data ...

Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation * Implement data quality … opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

data from internal systems and third‐party platforms including Salesforce, Sopra Banking Services, CACI and Experian. Engineer Modern Data Architectures - Design Lakehouse, Warehouse and Delta Lake solutions using medallion architectures to support enterprise analytics and reporting. Deliver Analytics-Ready Data Assets - Build trusted, well-modelled datasets and semantic … successfully built, enhanced or operated cloud‐based data platforms, ideally within Azure environments. Data Architecture Knowledge - You understand data modelling, ELT/ETL design, Delta Lake principles, data quality frameworks and modern engineering best practices. Cloud & DevOps Capability - You have experience with Azure services, automation, CI/ ...

building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline Implementing governance … with Azure Databricks Strong Python and PySpark, backed by advanced SQL Deep grounding in data warehousing, data modelling and integration patterns Unity Catalog and Delta Lake in a production environment Data quality and governance experience, ideally including Azure Purview Git, Agile delivery and a collaborative approach to code ...

considered for future opportunities that match your expertisePrimary Responsibilities• Client Engagement & Delivery• Data Pipeline Development (Batch and Streaming)• Databricks & Lakehouse Architectures• Data Modelling & Optimisation (Delta Lake, Medallion architecture)• Collaboration & Best Practices• Quality, Governance & SecurityBusiness Relationships• Solution Architects• Data Engineers, Developers, ML Engineers, and Analysts• Client stakeholders … mentoring teams of engineers to deliver high-quality scalable data solutions.Technical Expertise• Deep expertise with the Databricks platform (Spark/PySpark/Scala, Delta Lake, Unity Catalog, MLflow).• Proficiency in ETL/ELT tools such as DBT, Matillion, Talend, or equivalent.• Strong SQL and Python (or equivalent ...

PySpark. Develop ETL/ELT and data transformation workflows using Python and SQL. Create and maintain Databricks notebooks, jobs, workflows and clusters. Work with Delta Lake and lakehouse architectures. Monitor, troubleshoot and optimise pipeline performance. Maintain high standards of data quality and reliability. Work across AWS, Azure … years Data Engineering experience. Strong hands-on Databricks, PySpark, Python and SQL skills. Experience building and optimising ETL/ELT data pipelines. Knowledge of Delta Lake and data warehousing. Experience with at least one of AWS, Azure or GCP. Strong troubleshooting and problem-solving skills. A Computer Science ...