26 to 50 of 126 Delta Lake Jobs in London

/ELT solutions and enterprise data pipelines. Experience with event-driven and messaging architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture ...

Medallion Architecture, and ensuring our Databricks platform is secure, governed, and production ready. Key Responsibilities Design, build, and optimise scalable data pipelines using Databricks, Delta Live Tables, Synapse, and Delta Lake. Develop ingestion, transformation, and orchestration workflows across diverse sources (APIs, SQL, SFTP, SaaS platforms). Implement strong … clusters. Contribute to standards for versioning, DevOps, CI/CD, and testing for Databricks notebooks, workflows, and SQL assets. Support the rollout of Delta Sharing, Unity Catalog governance, and Iceberg/Delta interoperability where required. Partner with analysts and data scientists to ensure that high-quality, well‐modelled ...

Must Have At least 7 years as an Engineer in a Data Field Strong Azure data skills, including: Azure Data Factory V2 Azure Data Lake Storage V2 Azure Databricks Azure Function Apps& Logic Apps Azure Stream Analytics Azure Resource Manager skills (Terraform, Azure Portal, Az CLI and Az PowerShell … Strong PySpark, Delta Lake, Unity Catalog and Python skills. Includes ability to write unit and integration tests in Python with unittest, pytest, etc. Strong understanding of software development practices, such as SOLID principals, structuring code, testing, IOC, etc. Strong repositories, CI/CD skills. Strong knowledge of Kimball ...

including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster).Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg).Advanced SQL skills and proficiency in Python. Strong background in modern software development practices (testing ...

customer in Central London. Key Responsibilities Data Lakehouse Architecture: Design and optimize a high-performance lakehouse pattern utilizing Microsoft Fabric OneLake and Azure Data Lake Storage Gen2 (ADLS Gen2). Data Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers … OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark). Security & Compliance: Direct experience implementing security controls ...

native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch and streaming patterns Transformation and quality: dbt, Great ...

Unifeye. What You'll Do Design and build scalable batch and streaming data pipelines using Spark and Databricks. Develop modern Lakehouse architectures using Delta Lake and Medallion (Bronze, Silver and Gold) design principles. Build robust ingestion frameworks from APIs, files and streaming data sources. Ensure data quality, testing … data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good understanding of Lakehouse architecture and Delta Lake. Experience working with AWS, Azure or GCP. Experience implementing CI/CD, version control and deployment automation. Strong communication skills and experience working ...

workflows Excellent communication and collaboration skills in agile, cross-functional teams Ability to mentor junior engineers Nice to have: Experience with Databricks, PySpark, and Delta Lake Условия Competitive salary based on location, experience, and skills Comprehensive benefits package for employees and their families Continuous learning, structured development programs ...

Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation and quality:dbt, Great Expectations and equivalent ...

patterns, batch and streaming ingestion, event driven design Databricks and/or Microsoft Fabric at architecture level Azure Data Factory, Synapse, Azure SQL, Data Lake Storage Gen2 Advanced SQL, Python or PySpark, Spark optimisation, Delta Lake, Unity Catalog Azure DevOps or GitHub Actions, Terraform or Bicep Entra ...

Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment data. Analyse data lineage, dependencies, business logic and data quality to determine what should be migrated ...

Lead the assessment and modernisation of legacy SQL Server, ETL and reporting workloads into a Databricks-based architecture. Design scalable data models, Delta Lake structures and transformation pipelines supporting real estate investment data. Analyse data lineage, dependencies, business logic and data quality to determine what should be migrated ...

diagnosing and resolving issues Highly Desirable Skills Experience with streaming frameworks such as Flink, Kafka Streams and Kafka Connect Knowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data Catalog Exposure to DataOps practices and collaboration with Data Engineering teams Familiarity with GPU‐accelerated ...

systematic approach to diagnosing and resolving issuesHighly desirable skillsExperience with streaming frameworks such as Flink, Kafka Streams and Kafka ConnectKnowledge of modern data lake technologies including Delta Lake, Iceberg and Glue Data CatalogExposure to DataOps practices and collaboration with Data Engineering teamsFamiliarity with GPU-accelerated analytics using ...

Databricks Data Intelligence Platform, including migration from legacy data platforms, lakehouse modernisation, performance optimisation, governance, and production‐grade delivery. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data ...

delivery of complex Data & AI programmes, including planning, mobilisation, resourcing, financial management, delivery governance, risk and dependency management. Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, Lakehouse architecture, Databricks SQL, Workflows, Delta Live Tables, MLflow, Mosaic AI, Agent Bricks, Vector Search, and broader Data ...

Tech Environment My client is actively modernising their platform, moving away from legacy IaaS solutions and building a Databricks‐first Lakehouse . Spark, Delta Lake, CI/CD, and modern DevOps practices are central to how they design and scale data products. What You’ll Need Strong hands ...

reliability. Collaborate with analysts and AI engineers to deliver robust and reusable data products. Manage data lakes and warehouses using formats such as Delta Lake and Parquet. Implement best practices for data governance, performance, and security. Continuously evaluate and adopt new technologies to evolve Movar's data platform. ...

Strong understanding of data pipelines, ETL/ELT frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data Engineer ...

Alongside this, you'll be collaborative and customer-focused, with passion for continuous improvement and delivering high-quality data solutions. Experience with Python, Spark, Delta Lake, DevOps tools, agile methodologies or cloud data governance would be advantageous. In return, we are offering a salary of c.£60k. discretionary ...

Data Architect, Lead Data Architect, Principal Engineer or comparable senior technical leader. Advanced, hands-on Databricks experience in production environments, including lakehouse architecture, Delta Lake and relevant governance, workflow, SQL and optimisation capabilities. Proven ownership of modern data platforms or data products from architecture through production delivery ...

This is hands‐on delivery in a regulated iGaming environment: production pipelines, not notebooks. Key responsibilities Migrate ML data pipelines from Databricks (Spark/Delta Lake) to the SageMaker-based "golden template" architecture, working to the pattern set by the Lead ML Ops Engineer Build and amend feature … engineering pipelines, feature store integrations, and data access layers (S3, Glue, Lake Formation) supporting migrated models Implement parity and statistical testing to prove migrated pipelines/models match Databricks outputs Handle data migration/integration between Databricks and AWS: storage, permissions, IAM alignment Work within CI/ ...

This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company. Hybrid: London, Waterloo + Remote - 3 Days in the Office + 2 Days Remote Role Description: We are looking for a highly skilled Senior Data Engineer … scalable data pipelines and enabling real-time and batch analytics across the organization. Key Responsibilities: Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT) Develop and manage streaming data solutions for real-time analytics and decision-making Ingest data from various sourcesincluding SAP systems using ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise data architecture, balancing … growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions. ...

Senior Data Engineer, you'll take ownership of the design, development and optimisation of production-grade data solutions using Databricks, Apache Spark and Delta Lake. Your responsibilities will include: Designing, building and optimising scalable ETL/ELT data pipelines using Databricks, PySpark and Spark SQL Implementing Medallion Architecture patterns … Strong commercial experience with Databricks Excellent PySpark, Apache Spark and Spark SQL skills Experience building production-grade ETL/ELT pipelines Strong knowledge of Delta Lake and Lakehouse architecture Experience implementing Medallion/Bronze-Silver-Gold architectures Strong experience with Azure and/or AWS data platforms Knowledge ...