Databricks Architect

Job Title: Databricks Architect

Location: London, UK

Work Mode: Onsite

Experience: 10–15 Years (minimum 5+ years in Databricks)

Employment Type: Full-time

About the Role

We are seeking a highly experienced Databricks Architect to lead the design, development, and optimization of scalable data platforms and advanced analytics solutions. This role requires a deep understanding of Databricks Lakehouse architecture, big data engineering, cloud ecosystems, and enterprise data strategy. The ideal candidate will partner with business and technology leaders to deliver high-performance data systems that support analytics, AI/ML, and real-time insights.

Key Responsibilities

Architecture & Solution Design

• Architect end-to-end Databricks Lakehouse solutions including data ingestion, ETL/ELT pipelines, Delta Lake, data warehousing, and data governance frameworks.
• Define and enforce best practices, security standards, and performance optimization across Databricks workloads.
• Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog, and MLflow.
• Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks.

Data Engineering & Platform Build

• Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta.
• Oversee data quality frameworks, metadata management, lineage, and monitoring.

Leadership & Stakeholder Engagement

• Work closely with business leaders and product teams to translate requirements into robust technical solutions.
• Guide development teams, perform architecture reviews, and ensure platform engineering excellence.
• Conduct technical workshops, POCs, and roadmap planning for the Databricks environment.

Performance, Security & Governance

• Optimize Databricks clusters, query performance, and cost management.
• Implement data governance standards using Unity Catalog, RBAC, and compliance guidelines (GDPR, ISO).
• Ensure resilience, scalability, and disaster recovery readiness.

Required Skills & Qualifications

• 10–15 years overall experience in data engineering or data architecture.
• 5+ years of hands-on architecture experience specifically in Databricks.
• Expert in Apache Spark, PySpark, Databricks SQL, Delta Lake, and distributed systems.
• Strong understanding of Lakehouse architecture, BI ecosystems, and modern data platforms.
• Cloud expertise in Azure / AWS / GCP (Azure preferred for many UK clients).
• Experience with CI/CD, Infrastructure-as-Code (Terraform preferred), and DevOps.
• Proven experience leading teams and delivering enterprise-grade data solutions.

Preferred Qualifications

• Databricks certifications (e.g., Databricks Certified Data Engineer Professional, Architect).
• Experience building ML solutions using MLflow or integrating with cloud ML services.
• Experience in BFSI, Retail, Telecom, or Healthcare data environments.
• Experience with real-time data processing (Kafka, EventHub, Kinesis).

Soft Skills

• Excellent communication and stakeholder management skills.
• Strong problem-solving and solution-oriented mindset.
• Ability to lead in a fast-paced and dynamic environment.
• Collaborative, proactive, and able to provide strong technical direction.

Why Join Us?

• Work onsite with a high-impact, cutting-edge data team in London.
• Opportunity to lead large-scale data transformation initiatives.
• Exposure to enterprise data strategy, advanced analytics, and AI projects.
• Competitive compensation and career-growth opportunities.