20 of 20 Contract Delta Lake Jobs

building high-quality data pipelines and optimizing large-scale workloads on the Databricks Data Intelligence Platform. The ideal candidate is strong in PySpark, Delta Lake, Azure services, workflow automation, and performance tuning. Key Responsibilities Build and maintain scalable batch and streaming pipelines using Databricks notebooks, Jobs, and Workflows. … Optimize Spark and Delta Lake performance through cluster tuning, efficient configurations, AQE, and caching. Run performance tests and ensure workloads are cost-efficient and reliable. Apply data quality, lineage, and access controls using Unity Catalog and governance best practices. Develop reusable PySpark ETL and data transformation code. Manage ...

Azure & PySpark - SC Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing. Youll design and build scalable data pipelines and maintain high-quality, test-driven code … Python/PySpark pipelines for data ingestion, processing, and validation. Write unit and BDD tests using Behave, including mocking and patching. Create and optimize Delta Lake tables for reliable, performant data storage. Use Docker to manage consistent development, testing, and deployment environments. Build configurable, parameter-driven code ...

Design, develop, and optimize data pipelines using Azure Databricks and PySpark. Support data migration activities from legacy systems to modern cloud based architecture. Implement Delta Lake for efficient data storage and retrieval. Collaborate with data scientists and analysts to enable advanced economic modelling and forecasting. Ensure data security … . Strong hands on experience with Azure Databricks and Spark. Proficiency in PySpark, SQL, and Python (Scala desirable). Experience with Azure services: Data Lake, Synapse, Key Vault. Proven track record in ETL pipeline development and data migration. Familiarity with Delta Lake and data governance best practices. ...

Design, develop, and optimize data pipelines using Azure Databricks and PySpark. Support data migration activities from Legacy systems to modern cloud based architecture. Implement Delta Lake for efficient data storage and retrieval. Collaborate with data scientists and analysts to enable advanced economic modelling and forecasting. Ensure data security … . Strong hands on experience with Azure Databricks and Spark. Proficiency in PySpark, SQL, and Python (Scala desirable). Experience with Azure services: Data Lake, Synapse, Key Vault. Proven track record in ETL pipeline development and data migration. Familiarity with Delta Lake and data governance best practices. ...

Responsibilities Design, build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data. Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies. Conduct performance testing and cluster tuning to ensure cost-efficient, high-performing … Unity Catalogue and governance best practices. Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles. Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management. Integrate Databricks solutions with Azure services such as Azure ...

environments.Key Responsibilities Design, build, and maintain scalable data pipelines using Databricks Notebooks, Jobs, and Workflows for both batch and streaming data. Optimise Spark and Delta Lake performance through efficient cluster configuration, adaptive query execution, and caching strategies. Conduct performance testing and cluster tuning to ensure cost-efficient, high … Unity Catalogue and governance best practices. Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles. Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned data management. Integrate Databricks solutions with Azure services such as Azure ...

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable … validation pipelines using Python and PySpark. Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies. Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads. Build and manage containerised environments using Docker for consistent development, testing ...

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable … validation pipelines using Python and PySpark. Implement unit and behavior-driven testing with Behave, ensuring robust mocking and patching of dependencies. Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads. Build and manage containerised environments using Docker for consistent development, testing ...

umbrella inside IR35 Duration: Initial 3-month, subject to extension Hours: Full-time position Responsibilities include: Develop, optimise, and debug Databricks notebooks, Delta Lake workflows, and Genie-bot automations Design, maintain, and enhance ETL pipelines , ensuring reliability and performance Validate, transform, and orchestrate data using Python … engineer, document, and communicate existing data flows and pipeline structures for handover to internal teams Requirements: Strong hands-on experience with Databricks , including notebooks, Delta Lake, and Genie bots Proven background in designing and maintaining ETL pipelines , including debugging and optimisation Advanced Python skills for transformation, automation ...

optimize ETL/ELT workflows that ingest data from various on-prem and cloud-based sources. * Work with Azure services including Azure Data Lake Storage, Azure Data Factory, Azure Synapse Analytics, Azure SQL, and Event Hub. * Implement data quality validation, monitoring, metadata management, and governance processes. * Collaborate closely with … Required Skills and Experience * Proven experience as a Data Engineer working with Azure cloud services. * Strong proficiency in Databricks, including PySpark, Spark SQL, notebooks, Delta Lake, and job orchestration. * Strong SQL and data modeling skills (e.g., dimensional modeling, data vault). * Experience with Azure Data Factory or other ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines . Start Date: ASAP Contract Type: Outside IR35 Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner ...

workflows for performance and reliability. Essential Skills Proven experience with Databricks (including Spark-based data processing). Strong knowledge of Azure Data Platform (Data Lake, Synapse, etc.). Proficiency in Python and SQL for data engineering tasks. Understanding of data architecture and ETL processes. Ability to work independently … remote environment. Nice-to-Have Experience with CI/CD pipelines for data solutions. Familiarity with Delta Lake and ML pipelines . Start Date: ASAP Contract Type: Outside IR35 Apply Now: If you're a skilled Data Engineer looking for a flexible, remote opportunity with a Databricks Partner ...

optimise Data engineering solutions. Key requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with Azure … services: Data Lake Storage (Gen2), Key Vault, Azure Functions Familiarity with CI/CD for Databricks deployments Strong troubleshooting in distributed Data environments Excellent communication and collaboration skills Nice to have: Immediate availability Experience with enterprise-scale Databricks environments Knowledge of Azure Synapse, Power BI and cost optimisation strategies ...

optimise Data engineering solutions. Key requirements: Proven experience as a Data Engineer with Active Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with Azure … services: Data Lake Storage (Gen2), Key Vault, Azure Functions Familiarity with CI/CD for Databricks deployments Strong troubleshooting in distributed Data environments Excellent communication and collaboration skills Nice to have: Immediate availability Experience with enterprise-scale Databricks environments Knowledge of Azure Synapse, Power BI and cost optimisation strategies ...

platforms, supporting complex, large-scale workloads within mature Agile delivery environments. Deep hands-on expertise with Azure Databricks, including clusters, job orchestration, Workflows, Repos, Delta Lake, Unity Catalog, metastore configuration, and workspace administration. Strong experience with Azure DevOps YAML pipelines, Git branching strategies, artifact management, automated testing … Databricks resources, Azure networking, storage, key vaults, identities, private endpoints, and associated cloud infrastructure. Strong understanding of Azure cloud services such as Azure Data Lake Storage, Azure Key Vault, Azure Monitor, Azure VMs, Azure Networking, Azure Log Analytics, and Azure AD. Proficient in scripting with Python, Bash, or PowerShell ...

modular, test-driven design Experience with Behave for unit and BDD testing (mocking, patching) Proficiency in PySpark and distributed Data processing Solid understanding of Delta Lake (design and maintenance) Hands-on with Docker for development and deployment Familiarity with Azure services: Functions, Key Vault, Blob Storage Ability ...

modular, test-driven design Experience with Behave for unit and BDD testing (mocking, patching) Proficiency in PySpark and distributed Data processing Solid understanding of Delta Lake (design and maintenance) Hands-on with Docker for development and deployment Familiarity with Azure services: Functions, Key Vault, Blob Storage Ability ...

ready data solutions. Integrate Databricks with GCP/BigQuery to support cloud-native data engineering initiatives. Essential Skills Hands-on Databricks experience , including Spark, Delta Lake, and workflow orchestration. Strong SQL for complex querying and optimisation. Python proficiency , especially for data wrangling and automation. Financial domain knowledge , ideally ...

Responsibilities Analyze and document existing data pipelines, including diagrams and data-flow mapping. Develop and automate ETL pipelines using Python. Work with Databricks notebooks, Delta Lake, and Databricks Genie bots. Design, optimize, and debug ETL pipelines. Use SQL for querying, validation, and optimization in the Lakehouse environment. Person ...

Azure & PySpark - SC Cleared Contract £400-£458pd (Inside IR35) SC Clearance is Essential Summary Were looking for a Python Data Engineer skilled in PySpark, Delta Lake, Azure services, containerized development, and Behave-based testing click apply for full job details ...