1 to 25 of 37 Permanent Delta Lake Jobs in London

feeds honest, and ownership of the data warehouse and the data sync between our ERP and the wider estate. Our stack is Python, PySpark, Delta Lake and Databricks, with T‐SQL, Kimball dimensional modelling and Power BI on top, working closely with Microsoft Dynamics 365 Finance and Operations … stakeholders trust, from the models underneath through to well built, performant Power BI reports. Design and build data pipelines in Python, PySpark and Delta Lake, and write the T‐SQL behind our models and reconciliations. Keep an eye on cost, performance and reliability across the platform, and keep ...

build of large-scale data platforms using Azure Databricks and modern cloud technologiesImplement and optimize ETL workflows and streaming pipelines with PySpark and Delta Live Tables following Lakehouse principlesEnhance performance, manage cloud costs and ensure platform reliability for structured streaming workloadsDefine data governance, security and quality standards to maintain … including advanced optimization, data skew mitigation and query tuningStrong programming skills in Python with knowledge of modern software design principlesPractical experience with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake ...

Server and Databricks. Support Lakeflow pipelines, production jobs and data quality processes. Monitor, troubleshoot and optimise data workflows and platform performance. Work with Delta Lake, Unity Catalog and modern data lake architectures. Integrate Azure services including ADLS, Data Factory, Key Vault and REST APIs. Support CI/… enterprise environment. Excellent SQL Server/T-SQL and data engineering skills. Experience building and optimising large-scale data pipelines. Knowledge of Spark, Delta Lake, Unity Catalog and Azure data services. Exposure to Git, CI/CD and infrastructure automation tools. Strong communication skills and a proactive approach ...

data solutions, from conceptual and logical models through to physical implementation Building and maintaining scalable pipelines and lakehouse solutions using Azure Databricks, Spark and Delta Lake Designing and optimising Snowflake environments for performance, scalability and cost Supporting batch, streaming and real-time data processing, including high-volume transactional … environments Hands-on, current engineering skills: you still write the code, not just the diagrams Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake and Lakehouse architecture Strong Snowflake architecture and optimisation experience Experience with Azure Data Factory, ADLS Gen2, Synapse Analytics and/or Microsoft ...

enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics … Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business stakeholders and technical teams ...

model benchmarking methodologies Advanced programming skills in Python and SQL with strong knowledge of data modelling and ETL/ELT processes Solid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processing Experience with Azure Data Factory (ADF), Qlik Replicate ...

frameworks and model benchmarking methodologiesAdvanced programming skills in Python and SQL with strong knowledge of data modeling and ETL/ELT processesSolid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processingExperience with Azure Data Factory (ADF), Qlik Replicate ...

cost and usage data platform on Databricks, ingesting Azure Cost Management exports, reservation and Savings Plan data, resource inventory and tagging metadata into Delta Lake and modelling it into trusted datasets. Build and own Power BI semantic models behind cost reporting, including DAX measures, composite and DirectQuery designs … rather than simply creating reports. Nice to HaveFinOps or cloud-cost experience, including Azure Cost Management exports, FOCUS, reservations or Savings Plans. Databricks and Delta Lake. KQL.TMDL and XMLA.PowerShell. Experience with cloud cost optimisation, allocation/showback or cost attribution. What We're UsingPython — production services, data pipelines, automation ...

highly visible to senior leadership. You Will Work With Designing and scaling Azure Databricks-based data platforms Building robust data pipelines using Databricks, PySpark, Delta Lake and Airbyte Developing AI and LLM-powered capabilities within customer-facing products Delivering data products that support commercial decision-making Optimising Azure … Advanced Python, SQL and PySpark skills Experience designing and building modern data platforms and pipelines Databricks optimisation, performance tuning and cost management Experience with Delta Lake and Lakehouse architectures Strong Azure cloud experience Experience with modern data ingestion tools such as Airbyte Exposure to Generative AI, LLMs ...

customer in Central London. Key ResponsibilitiesData Lakehouse Architecture: Design and optimize a high-performance lakehouse pattern utilizing Microsoft Fabric OneLake and Azure Data Lake Storage Gen2 (ADLS Gen2).Data Zone Management: Maintain architectural segregation between raw storage (immutability, landing) and curated data zones (cleansed, conformed business layers) while securely … ecosystem (OneLake, Data Factory, Fabric Lakehouse).Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL) and Python (PySpark).Security & Compliance: Direct experience implementing security controls ...

native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch and streaming patterns Transformation and quality: dbt, Great ...

deadlines. Databases & Tools knowledge: Microsoft SQL Server (administration, T-SQL, DML, DDL, SSIS), Databricks, Power BI.Experience in Azure Technologies: ADLS, ADF, Azure SQL Database, Delta Lake, Azure DevOps, Logic Apps, Function Apps, Key Vaults, Storage Accounts. Infrastructure & Automation: Terraform, Python/PySpark. Expertise in data modeling, integration ...

disciplines within enterprise environments4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse PlatformStrong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practicesExperience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines for scalable analytics and artificial … Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologiesBuild, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta LakeDevelop governed datasets and data products that support reporting, analytics, and artificial intelligence initiativesPartner with business stakeholders and technical teams to translate requirements into ...

level discussion and technical detail Hands on implementation experience, ideally from a data engineering or developer background Databricks at architecture level on Azure, including Delta Lake, Unity Catalog and Spark optimisation Lakehouse and medallion patterns, batch and streaming ingestion, event driven design Conceptual, logical and physical data modelling. … Dimensional and data vault both useful Advanced SQL, Python or PySpark Declarative pipelines (Delta Live Tables or equivalent), Databricks Workflows, Auto Loader and Structured Streaming Microsoft Purview for data governance, cataloguing and lineage Azure Data Factory, Azure SQL, Data Lake Storage Gen2 Azure DevOps or GitHub Actions, Databricks ...

extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise data architecture, balancing … growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ELT solutions. ...

aggregation patterns) Strongly Preferred Experience with Azure landing zones and CAF (Cloud Adoption Framework) Microsoft Purview for data governance and cataloging Experience with Delta Lake, Spark optimization, data quality frameworks Azure networking certifications or equivalent deep knowledge Container orchestration (AKS) API design and management (API Management, App Gateway ...

Office + 2 Days RemotePay: c.700 per day PAYEConsultant: Joe MooreThis is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company. Role Description: We are looking for a highly skilled Senior Data Engineer with strong analytical capabilities … scalable data pipelines and enabling real-time and batch analytics across the organization. Key Responsibilities: Design and implement scalable data pipelines using Databricks and Delta Live Tables (DLT)Develop and manage streaming data solutions for real-time analytics and decision-makingIngest data from various sourcesincluding SAP systems using Qlik ...

Databricks Define and enforce platform standards, ensuring compliance with data governance and security policies Build and optimize Databricks streaming workloads, including Structured Streaming and Delta Live Tables Collaborate with cross-functional teams to translate requirements into robust technical solutions Drive platform performance tuning, cost optimization and monitoring strategies … 3+ years in a tech lead capacity Strong hands-on expertise in Databricks for enterprise-scale workloads Proficiency in PySpark, Spark Structured Streaming, and Delta Lake Solid programming skills in Python and demonstrated SQL performance optimization experience Deep understanding of Azure Data Platform services and cloud-native engineering ...

scalable, secure and high-performing solutions that support analytics, AI, and enterprise reporting. Key Skills Required AWS (S3, Glue, Redshift, Athena, Kinesis, Lambda) Data Lake & Lakehouse architectures ETL/ELT & data pipelines Python, SQL & PySpark Data modelling (Kimball, Data Vault) Data governance & security Terraform/CloudFormation Real-time & batch … best practices Working closely with business and technical stakeholders Leading data transformation and cloud migration initiatives Desirable Databricks, Snowflake, dbt Kafka, Apache Iceberg, Delta Lake Data Mesh/Data Fabric AI & ML-enabled data platforms ...

interpersonal skills - you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environmentExperience in the financial markets, especially in delta one, store of value, and/or FICC options tradingExperience with Linux-based, concurrent, high-throughput, low-latency software systemsExperience with pipeline orchestration frameworks … e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about DRW's processing activities ...

Skills & Experience Commercial experience within Data Engineering and cloud-based data platforms. Strong hands-on experience with the Databricks Lakehouse Platform. Solid understanding of Delta Lake, Medallion Architecture and modern data engineering principles. Experience designing and delivering enterprise-scale data solutions. Strong Python development skills. Experience working with ...

agile delivery environments. Strong communication skills and ability to interact efficiently with various stakeholder groups including business users. Nice to haveExperience with Databricks, Spark, Delta Lake, or similar modern data engineering platforms. Experience designing enterprise data lakes, lakehouses, or large-scale analytical platforms. Domain experience within Investment Management ...

Demonstrable data modelling experience, including practical experience with medallion architecture, and comfortable contributing technical input into modelling discussions alongside IT Experience with Databricks, including Delta Lake; familiarity with Unity Catalog and semantic layer/consumption-layer tooling is a plus (Power BI & Databricks Dashboards) Power BI experience, able ...

PySpark and experience optimising large-scale distributed processingBig data experience — designing and operating systems for datasets and workloads well beyond single-machine scaleApache Iceberg, Delta Lake, or similar open table formats and experience with lakehouse architectures, Databricks, Snowflake, or comparable data platformsCI/CD tooling such as GitLab ...

/CD.The ability to understand customer requirements and translate them into practical technical solutions. Excellent analytical, problem-solving and communication skills. Desirable experienceData-lake or lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Snowflake or AWS Lake Formation. Image or video visualisation, enhancement ...