19 of 19 Delta Lake Jobs in London

their own workflows. We are in the process of a major technology migration, moving our core data pipelines from generating HDF5 files into generating Delta Tables. This will improve efficiency, ensure future scalability and modernise the architecture. We are looking for an experienced lead to head up a small … cutting‐edge FinTech company with the potential to grow your technical, finance and cloud‐ & data‐engineering skills. Responsibilities Technical leadership and management of the delta lake migration team, organising work into 2‐week sprints, upwards reporting on progress and blockers. Development, testing and migration of some ...

implement scalable Databricks Lakehouse platforms on AWS and/or Azure aligned to client requirements Architect end-to-end data platforms including ingestion, storage (Delta Lake), processing, and consumption layers Build and configure cloud infrastructure using infrastructure-as-code (e.g. Terraform & Declarative Automation Bundles(DAB's)) Establish secure … pipelines, and release processes for Databricks deployments Implement monitoring, logging, cost optimisation, and performance tuning across the platform Design and implement data pipelines using Delta Live Tables, Auto Loader, and Databricks Workflows for both batch and streaming workloads Client Delivery & Enablement responsibilities: Work directly with clients to translate business ...

practices, and guiding engineering teams to deliver scalable, secure, and cost optimized data platforms. \* Design end to end data platforms leveraging Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and ML/AI workloads. \* Define data ingestion, transformation, governance, and consumption frameworks aligned to enterprise standards. \* Evaluate and recommend … emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated, governed data ...

DevOps, Power BI, Fabric workspaces AWS cloud, S3, cross-cloud connectivity, API Gateways GCP (BigQuery, GCS, Dataflow) Deep expertise in: Databricks (Lakehouse architecture, Delta Lake, Unity Catalog, Delta Sharing, Pyspark, Spark SQL etc.) Snowflake (data warehousing & data sharing, Iceberg, ) Strong understanding of: Data architecture patterns, industry benchmarking ...

term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics ...

reliability. Collaborate with analysts and AI engineers to deliver robust and reusable data products. Manage data lakes and warehouses using formats such as Delta Lake and Parquet. Implement best practices for data governance, performance, and security. Continuously evaluate and adopt new technologies to evolve Movar's data platform. ...

modern data platforms (Databricks, Snowflake, Synapse, BigQuery) Ensure robust data engineering practices (Airflow, dbt, Prefect) and governance (Collibra, Purview) Drive scalable lakehouse architectures (Delta Lake, Iceberg, Hudi) Cloud, Infrastructure & MLOps Champion multi-cloud engineering across Azure, AWS, and GCP Lead MLOps strategy (MLflow, Kubeflow, SageMaker, Vertex ...

data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/AWS). SQL Database ...

data engineering experience — proven track record building big data processing platforms in production, moving beyond scripting to robust engineering solutions (e.g., Databricks/Delta Lake, Snowflake, BigQuery). Hands-on experience architecting Data Warehouses and Data Lakes. API and service design maturity — experience designing multi-service systems with ...

data governance frameworks, metadata management, and regulatory compliance It would be a bonus if you have: Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, or data observability tooling Experience leading architecture workshops with both technical and non-technical stakeholders A background providing technical mentorship ...

experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory • Hands-on data engineering experience (Spark/PySpark, Delta Lake) • Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC • Experience leading Agile teams and managing risks, dependencies, and senior ...

Catalog grants, schema mismatches and query failures. Technical Environment : Palantir Foundry, Foundry Virtual Tables, Ontology, Workshop, Pipeline Builder, Code Repositories, AWS, Databricks, Terraform, IaC, Delta Lake, Parquet, Iceberg, CSV, JSON, Python, PySpark, SQL, CI/CD, GitLab, GitHub Actions. ...

medallion architecture) Leading the design of cloud data platforms (AWS) – including services like Redshift, S3, Glue, Athena Evaluating and introducing new technologies (e.g. Iceberg, Delta Lake, observability tools) Driving data governance, standards, and best practice Acting as a trusted advisor to both technical and business stakeholders What ...

Docker, and Kubernetes Familiarity with One of the following programming languages: Elixir, Python, Java, C#, C++, Rust One of the following data formats: Parquet, Delta Lake, Iceberg Pricing, risk management, and portfolio analytics Exchange-traded financial assets, statistics, or financial engineering Data structures and design/analysis ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. For more information, follow Databricks on Twitter, LinkedIn and Facebook. Benefits: At Databricks, we strive to provide comprehensive benefits and perks ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. To learn more, follow Databricks on Twitter, LinkedIn and Facebook. Benefits At Databricks, we strive to provide comprehensive benefits and perks ...

BMLL Technologies Limited in Greater London is seeking an experienced Technical Lead to manage the migration to Delta Lake. This role combines hands-on development, primarily in C++ and Python, with team leadership in Agile sprints. You will guide a small team through the migration process, collaborating with architects ...

BMLL Technologies is seeking a Lead Developer to oversee a pivotal technology migration. The role involves hands-on development primarily in C++ and Python, along with managing a small team in Agile sprints. You will ...