76 to 100 of 178 Delta Lake Jobs in the UK

data-protection focus throughout What you'll need: 5+ years' progressive data engineering experience Strong Databricks and Spark expertise - platform internals, performance, DAB, Delta Lake, declarative pipelines Advanced Python, with solid software/DevOps practice (version control, unit testing, pipelines-as-code, CI/CD) Excellent data modelling ...

delivery, and enhance solution quality while applying sound engineering judgement and responsible AI practices. What You'll DoDesign and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services. Build real-time, event-driven solutions using technologies such as Kafka and Azure ...

OAuth 2.0, Azure Key Vault, and role-based access control (RBAC). Awareness of modern lakehouse concepts and integration with layered data platforms, including Delta Lake and raw/curated data layers. Exposure to Spark/Databricks and PySpark or Spark SQL for collaborating with data engineering teams ...

skills – you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environment Experience in the financial markets, especially in delta one, store of value, and/or FICC options trading Experience with Linux‐based, concurrent, high‐throughput, low‐latency software systems Experience with pipeline … orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information about DRW’s processing activities ...

interpersonal skills - you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environmentExperience in the financial markets, especially in delta one, store of value, and/or FICC options tradingExperience with Linux-based, concurrent, high-throughput, low-latency software systemsExperience with pipeline orchestration frameworks … e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experienceFor more information about DRW's processing activities ...

skills - you communicate clearly with stakeholders as well as other engineers, fostering a collaborative, supportive working environment Experience in the financial markets, especially in delta one, store of value, and/or FICC options trading Experience with Linux-based, concurrent, high-throughput, low-latency software systems Experience with pipeline … orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelor or advanced degree in Computer Science, Mathematics, Statistics, Physics, Engineering, or equivalent work experience For more information about DRW's processing activities ...

/column-level security, lineage and audit Build reusable Terraform modules for Databricks and automate platform changes through CI/CD pipelines Tune Spark, Delta and Photon workloads to improve performance and reduce DBU and cloud spend Manage identity, SSO/SCIM, service principals, secret scopes and network isolation … meet security and compliance needs Operate Databricks Workflows, Delta Live Tables and orchestration integrations used by Data Scientists and Data Support MLflow, model serving and feature store usage for predictive modelling and GenAI work Monitor jobs, clusters and warehouses; lead incident response and post-incident reviews QA own work ...

have the opportunity to: ·Design and build production-grade pipelines using Azure Databricks, Python, PySpark and SQL ·Develop scalable Lakehouse architecture using Delta Lake and Unity Catalog ·Build data and BI products from scratch and own them through to implementation ·Connect marketing, sales and operational data to create ...

requirements into technical solutions and working across technical and non‐technical teams. “Nice To Have” Skills and Experience Experience with Databricks, Spark/PySpark, Delta Lake or comparable technologies. Experience with Celonis, process mining or process intelligence. Experience with enterprise platforms such as Salesforce, SAP or ServiceNow. Exposure ...

time streaming, and CI/CD pipelines (Git, Terraform, Databricks Asset Bundles). Deep knowledge of Databricks/Snowflake environments, including Unity Catalog, workspaces, Delta Lake, and networking/security (VPNs, Private Links, Firewalls) and/or Snowflake Platform including Cortex, Horizon and Snowpark. Advanced proficiency in Azure ...

time streaming, and CI/CD pipelines (Git, Terraform, Databricks Asset Bundles). Deep knowledge of Databricks/Snowflake environments, including Unity Catalog, workspaces, Delta Lake, and networking/security (VPNs, Private Links, Firewalls) and/or Snowflake Platform including Cortex, Horizon and Snowpark. Advanced proficiency in Azure ...

time streaming, and CI/CD pipelines (Git, Terraform, Databricks Asset Bundles). Deep knowledge of Databricks/Snowflake environments, including Unity Catalog, workspaces, Delta Lake, and networking/security (VPNs, Private Links, Firewalls) and/or Snowflake Platform including Cortex, Horizon and Snowpark. Advanced proficiency in Azure ...

data concepts, including storage, processing, and modelling. Core requirements include:Databricks - hands-on experience building and managing workloads on the Databricks platform, including Delta Lake, Unity Catalog, and Declarative Automation BundlesPySpark - proficiency in writing and optimising PySpark code for data transformation and processingAzure - solid experience working within … Azure ecosystem, including services such as Azure Data Factory, Azure Data Lake Storage, and Azure DevOpsAzure SQL Server - experience with Azure SQL Server is important, as it forms the foundation of our existing data warehouse. You will be working across both our SQL Server environment and our Databricks platform ...

reviews and CI/CD.The ability to understand customer requirements and translate them into practical technical solutions.Excellent analytical, problem-solving and communication skills.Desirable experienceData-lake or lakehouse technologies such as Delta Lake, Apache Iceberg, Apache Hudi, Databricks, Snowflake or AWS Lake Formation.Image or video visualisation, enhancement ...

divergent source systems into canonical models across multiple products. Strong grounding in Lakehouse architecture trade-offs: table formats, query engines, ingestion patterns (Iceberg vs Delta; Dremio vs Trino vs Snowflake; streaming vs batch). Hands-on experience with: Dremio, Snowflake, or Databricks AWS data services (S3, Glue, Athena, Lake … Stream processing (Apache Flink, Spark Structured Streaming, or equivalent) SQL and query optimisation at scale Data and table formats (Parquet, Avro, Apache Iceberg or Delta Lake) Experience designing SaaS data models, tenant isolation, shared schemas, access patterns, at scale. Ability to communicate architecture decisions to engineers, product leadership ...

This is an exciting role for a Senior Data Engineer with strong Databricks and Delta to join a world-leading Energy Trading company. Role Description: We are looking for a highly skilledSenior Data Engineerwith strong analytical capabilities and hands-on experience in building scalable data solutions. The ideal candidate … implementing scalable data pipelines and enabling real-time and batch analytics across the organization. Key Responsibilities: Design and implement scalable data pipelines usingDatabricks and Delta Live Tables (DLT) Develop and managestreaming data solutionsfor real-time analytics and decision-making Ingest data from various sourcesincludingSAP systems using Qlik ReplicateandADFto ingest ...

implementation of scalable data architectures on the Databricks Lakehouse platform. Oversee the development of efficient ETL/ELT pipelines using PySpark, SQL, and Delta Lake. Mentor and guide junior data engineers, promoting best practices in code quality, CI/CD, and performance tuning. Collaborate with data architects, analysts … London office for 3 days per week for collaborative working and stakeholder engagement. Essential Requirements & Skills Databricks Expertise: Extensive hands-on experience with Databricks, Delta Lake, Unity Catalog, and Spark optimization. Programming Languages: Advanced proficiency in Python/PySpark and SQL. Cloud & Infrastructure: Strong background in cloud data ...

maintaining scalable data pipelines using Databricks and PySpark. Developing high-performance batch and streaming data solutions. Designing and implementing modern Lakehouse architectures using Delta Lake and medallion design principles. Optimising Databricks workloads for performance, reliability and scalability. Implementing data quality, monitoring, and observability frameworks. Supporting governance, security … engineers with strong hands-on Databricks experience. Essential Skills Commercial experience building solutions within Databricks. Strong hands-on expertise with PySpark, Spark SQL, and Delta Lake. Experience designing and supporting production-grade ETL/ELT pipelines. Strong Python and SQL development skills. Experience working with large-scale distributed data ...

maintaining scalable data pipelines using Databricks and PySpark. Developing high-performance batch and streaming data solutions. Designing and implementing modern Lakehouse architectures using Delta Lake and medallion design principles. Optimising Databricks workloads for performance, reliability and scalability. Implementing data quality, monitoring, and observability frameworks. Supporting governance, security … engineers with strong hands-on Databricks experience. Essential Skills Commercial experience building solutions within Databricks. Strong hands-on expertise with PySpark, Spark SQL, and Delta Lake. Experience designing and supporting production-grade ETL/ELT pipelines. Strong Python and SQL development skills. Experience working with large-scale distributed data ...

Senior Data Engineer – Databricks & DLT (Delta Live Tables) Certain Advantage are recruiting on behalf of our trading client for a Senior Data Engineerwith strong analytical capabilities and hands‐on experience in building scalable data solutions using a programming language like Python or Scala as well as experience in Databricks … Delta Live Tables - This experience is essential. Suitable candidates will have expertise inDatabricks,streaming architectures, data ingestion frameworks, anddashboarding, along with experience working withSAP data sources. You will be responsible for designing and implementing scalable data pipelines and enabling real‐time and batch analytics across the organization. Your Responsibilities ...

data governance frameworks, metadata management, and regulatory compliancIt would be a bonus if you have:Familiarity with emerging technologies such as Apache Iceberg, Delta Lake, real-time/streaming architectures, or data observability toolingExperience leading architecture workshops with both technical and non-technical stakeholdersA background providing technical mentorship ...

with business stakeholders and understand the broader context behind technical requirements. Nice-to-Have Skills Experience with Apache Iceberg or similar table formats (e.g., Delta Lake, Hudi). Familiarity with CI/CD platforms such as GitLab CI, Jenkins, or GitHub Actions. Exposure to data quality frameworks such ...

PySpark. You write reusable, parameterized functions and work comfortably with JSON, CSV, and Parquet. Platform knowledge Deep experience with Databricks (Declarative Pipelines, DABS, Delta Lake, Spark optimization, job orchestration). Familiarity with the Azure ecosystem. Working understanding of Unity Catalog. About you Around 6 or more years ...

## **Mandatory Skills****Azure Databricks Platform Engineering*** Hands-on implementation and troubleshooting of Azure Databricks* Databricks Serverless configuration and workload optimisation* Databricks SQL and Delta Lake development* Cluster management, compute policies, autoscaling, and workload monitoring**Data Engineering*** Strong Python, PySpark, and SQL development skills* Design and build ingestion ...

## **Mandatory Skills****Azure Databricks Platform Engineering*** Hands-on implementation and troubleshooting of Azure Databricks* Databricks Serverless configuration and workload optimisation* Databricks SQL and Delta Lake development* Cluster management, compute policies, autoscaling, and workload monitoring**Data Engineering*** Strong Python, PySpark, and SQL development skills* Design and build ingestion ...