Senior Azure Data Engineer (Databricks + GenAI)

Role: Senior Azure Data Engineer (Databricks + GenAI)

Location: UK Remote

Engagement: Permanent

Overview:

We are seeking a Senior Azure Data Engineer to take ownership of the data engineering capability across multiple enterprise-scale Azure Databricks programmes. This role sits at the intersection of data engineering, cloud architecture, and GenAI enablement, building production-grade lakehouse platforms that power analytics, automation, and AI-driven applications. You will act as a technical lead and client-facing engineer, responsible for designing, delivering, and governing scalable Azure + Databricks data platforms.

Key Responsibilities:

1. Architecture Leadership

• Define Azure Databricks lakehouse architectures (medallion model)
• Review and approve engineering designs across projects

2. Data Engineering Delivery

• Build end-to-end pipelines using:
• Azure Databricks (PySpark, Delta Lake, Workflows)
• Azure Data Factory / Synapse Pipelines
• ADLS Gen2
• Deliver curated data products (bronze / silver / gold)

3. GenAI Enablement

• Prepare data for RAG and AI workloads
• Work with Azure OpenAI + Azure AI Search
• Implement embedding, chunking, and retrieval strategies

4. CI/CD & Infrastructure

• Implement IaC using Terraform or Bicep
• Manage CI/CD pipelines using Azure DevOps or GitHub Actions
• Use Databricks Asset Bundles for deployment automation

5. Governance & Security

• Implement Unity Catalog, Purview, RBAC, and data lineage
• Ensure secure-by-design architecture (Key Vault, Entra ID)

6. Client Engagement

• Lead workshops, design sessions, and technical discussions
• Translate business requirements into structured engineering delivery

7. Mentorship

• Guide and support mid-level engineers
• Enforce engineering standards and best practices

Required Experience:

• 8+ years in data engineering
• 4+ years hands-on Azure data platform experience
• Expert-level Azure Databricks (PySpark, Delta Lake, Jobs, tuning)
• Strong Azure Data Factory / ADLS Gen2 / Azure SQL experience
• Solid Python and SQL engineering background
• Experience with lakehouse / medallion architecture
• CI/CD & IaC (Terraform or Bicep + Azure DevOps/GitHub Actions)
• Understanding of Azure AI Search / Azure OpenAI / RAG concepts
• Experience with Unity Catalog / Purview / data governance frameworks

Soft Skills:

• Strong client-facing communication skills
• Confident leading technical design discussions
• Ability to manage ambiguity and define structure
• Mentoring and leadership capability

Nice to Have:

• Power BI / Fabric experience
• Salesforce data integration experience
• Microsoft certifications (DP-203, DP-700, AZ-305, etc.)
• Experience in real-time or operational data environments
• Exposure to Data/AI Centre of Excellence environments