Lead Data Platform Engineer - Databricks - IAC - Terraform - Azure Data Factory - Data Lakehouse

Lead Data Platform Engineer - Databricks - IAC - Terraform - Azure Data Factory - Data Lakehouse

The Data Platform Engineer designs, develops, automates, and maintains secure, scalable, and compliant data platforms that enable the firm to efficiently manage, analyse, and utilise data. The role ensures that data solutions are robust and reliable while meeting regulatory obligations and safeguarding client confidentiality.

Key Responsibilities

• Design and architect scalable, secure, and compliant data platforms and solutions, producing technical documentation and securing approvals through governance bodies such as Architecture Review Boards.
• Build and deliver robust data solutions using Databricks, PySpark, Spark SQL, Azure Data Factory, and Azure services.
• Develop APIs and write efficient Python, PySpark, and SQL code to support data integration, processing, and automation.
• Implement and manage CI/CD pipelines and automated deployments using Azure DevOps to enable reliable releases across environments.
• Develop and maintain infrastructure-as-code (eg, Terraform, ARM) to provision and manage cloud resources, including ADF pipelines, Databricks assets, and Unity Catalog components.
• Monitor, troubleshoot, and optimise data platform performance, reliability, and costs, identifying bottlenecks and recommending improvements.
• Create dashboards and observability tools to report on platform performance, usage, incidents, and operational KPIs.

Knowledge, Skills & Experience

• Degree in Computer Science, Data Engineering, or a related field.
• Proven experience designing and building cloud-based data platforms, ideally within Azure.
• Strong hands-on expertise with Databricks, PySpark, Spark SQL, and Azure Data Factory.
• Solid understanding of Data Lakehouse architecture and modern data platform design.
• Proficiency in Python for data engineering, automation, and data processing.
• Experience developing and integrating REST APIs for data services.
• Strong DevOps experience, including CI/CD, automated testing, and release management for data platforms.
• Experience with Infrastructure as Code tools such as Terraform or ARM templates.
• Knowledge of data modelling, ETL/ELT pipelines, and data warehousing concepts.
• Familiarity with monitoring, logging, and alerting tools (eg, Azure Monitor).

Desirable

• Experience with additional Azure services (eg, Fabric, Azure Functions, Logic Apps).
• Knowledge of cloud cost optimisation for data platforms.
• Understanding of data governance and regulatory compliance (eg, GDPR).
• Experience working in regulated or professional services environments.