4 of 4 Remote/Hybrid Contract Delta Lake Jobs

patterns, batch and streaming ingestion, event driven design Databricks and/or Microsoft Fabric at architecture level Azure Data Factory, Synapse, Azure SQL, Data Lake Storage Gen2 Advanced SQL, Python or PySpark, Spark optimisation, Delta Lake, Unity Catalog Azure DevOps or GitHub Actions, Terraform or Bicep Entra ...

improving security, supportability and delivery speed. 4. Data Engineering and Integration Design and build reliable ingestion and transformation pipelines using Python, PySpark, SQL and Delta Lake. Implement full and incremental ingestion, CDC where appropriate, schema evolution, reconciliation, error handling and data quality controls. Design reusable integration patterns for REST … Azure Databricks implementation and troubleshooting Databricks Serverless and compute/workload optimisation Azure identity, networking, security, secrets, monitoring and private connectivity Databricks SQL, Delta Lake and performance optimisation Python, PySpark and SQL Jobs/workflows, incremental processing, CDC and data quality REST/API and external data integration ...

implementation of scalable data architectures on the Databricks Lakehouse platform. Oversee the development of efficient ETL/ELT pipelines using PySpark, SQL, and Delta Lake. Mentor and guide junior data engineers, promoting best practices in code quality, CI/CD, and performance tuning. Collaborate with data architects, analysts … London office for 3 days per week for collaborative working and stakeholder engagement. Essential Requirements & Skills Databricks Expertise: Extensive hands-on experience with Databricks, Delta Lake, Unity Catalog, and Spark optimization. Programming Languages: Advanced proficiency in Python/PySpark and SQL. Cloud & Infrastructure: Strong background in cloud data ...

observability Knowledge of Auto Loader, Lakeflow Connect and batch, streaming or incremental ingestion patterns Production-level PySpark and SQL development skills Strong knowledge of Delta Lake, including MERGE, schema evolution, OPTIMIZE, ZORDER and VACUUM Experience with Databricks Workflows, orchestration and CI/CD Understanding of monitoring, alerting, replay ...