Databricks Solution Architect
Consultancy client are looking to recruit a Databricks Solution Architect to define and deliver enterprise-scale data platform architectures using Azure Databricks.
Candidates must possess deep hands-on Databricks engineering knowledge alongside experience designing secure, scalable and highly available data platforms for UK Public Sector organisations. This is a technical architecture role requiring the ability to move between architecture design, technical implementation and engineering leadership.
Essential Technical Skills
- 5+ years of hands-on Databricks platform experience.
- Extensive Azure Databricks architecture experience.
- Expert-level PySpark, Spark SQL and Python knowledge.
- Deep understanding of Apache Spark processing architecture.
- Extensive Delta Lake and Lakehouse architecture experience.
- Strong experience designing Medallion Architecture.
- Expert knowledge of Unity Catalog, governance, permissions and data lineage.
- Experience designing Databricks workspaces, clusters, compute policies and networking.
- Knowledge of private endpoints, managed identities, Key Vault and secure connectivity.
- Experience with Azure Data Lake Storage Gen2, Azure Data Factory and Azure services.
Public Sector Experience
- Extensive experience delivering enterprise data architectures
- Understanding of UK Government security principles and data governance requirements.
- Experience architecting platforms handling sensitive and restricted government data.
- Knowledge of UK GDPR, Data Protection Act 2018 and relevant government security standards.
- Experience producing architecture documentation, HLDs, LLDs and technical design authorities.
- Experience working with public sector enterprise architecture and governance boards.
Security Clearance Requirements
- Be eligible and willing to undergo SC clearance
- Previous experience working on sensitive government projects is advantageous.
Responsibilities
- Own the end-to-end Databricks solution architecture.
- Define scalable and secure Lakehouse architecture.
- Establish engineering and architecture standards.
- Design data ingestion, transformation and consumption patterns.
- Define Unity Catalog and enterprise data governance models.
- Provide technical direction to data engineering teams.