Databricks Engineer
- Location
- Greater London, England, United Kingdom
streaming data pipelines using Azure Databricks, Apache Spark, PySpark, Python, SQL, and Delta Lake. Develop reusable ETL and ELT frameworks for data ingestion, transformation, validation, and publishing across lakehouse layers. Design and manage Delta Tables, including schema evolution, data quality controls, reliability, and performance optimization. Implement data governance, access control … experience designing and operating production-grade ETL or ELT pipelines on Azure. Hands-on experience implementing CI/CD pipelines using GitLab, including automated validation and multi-environment deployments. Working knowledge of Azure Data Lake Storage, Azure Data Factory, and Azure DevOps. Demonstrated ability to tune Spark workloads and troubleshoot ...