Databricks Product Architect - must be SC cleared

Databricks Product Architect - must be SC cleared - London - 1-2 days and remote - £640 - 682

Minimum five years of experience in data architecture roles, with a specific focus on data product design and delivery.

• Strong, demonstrable expertise in Databricks, including Lakehouse architecture, Delta Lake, Unity Catalog, and Databricks SQL.
• Proven experience designing and delivering self-service data platforms, including data cataloguing, access management, and data contracts.
• Deep understanding of data modelling, schema design, and the management of structured and semi-structured data at scale.
• Experience working within UK public sector organisations, with a clear understanding of the associated governance, compliance, and stakeholder complexities.
• Familiarity with data security frameworks relevant to government, including Government Security Classifications and GDPR.
• Strong grasp of data mesh principles and their practical application in large-scale organisations.
• Experience with cloud infrastructure in AWS, including integration patterns with Databricks on AWS (eg S3, IAM, VPC, Glue).
• Excellent communication skills, with the ability to engage both technical teams and senior stakeholders effectively.

Role

Data Product Architecture & Design

• Define and lead the architectural design of data products built on a Databricks Lakehouse platform within a complex public sector environment.
• Establish data product standards, patterns, and templates that ensure consistency, reusability, and interoperability across teams.
• Design data products to be self-describing, versioned, and consumer-ready - aligning to data mesh principles where applicable. Examples include a Person Entity Product providing a curated, deduplicated view of individuals across source systems, or a Passenger Movement Product structuring border crossing events for Real Time and batch consumers.
• Ensure all data products are optimised for performance, scalability, and cost-efficiency within the Databricks environment.

Self-Service Platform Enablement

• Architect and deliver a self-service data platform layer that allows consumers to discover, access, and use data products independently.
• Define access control patterns, data contracts, and SLAs for published data products - for example, ensuring a Casework Performance Product serving operational dashboards has clearly defined refresh SLAs and consumer-facing schemas.
• Work closely with platform and infrastructure teams to ensure the underlying Databricks environment supports self-service consumption patterns.
• Champion the adoption of Unity Catalog for data governance and access management across the platform, including the management of reference and lookup products such as Address & Geography or Document Validity datasets.

Security, Governance & Compliance

• Ensure all data products meet client and cross-government data security and governance standards.
• Apply data classification, access controls, and audit logging in line with public sector information assurance requirements - particularly for sensitive products such as Immigration Status or Watchlist Matching data, where strict access tiering and full audit trails are essential.

Desirable Skills & Experience

• Databricks Certified Data Engineer Professional or equivalent Databricks certification.
• Experience with adjacent data tooling such as dbt, Apache Spark, or AWS Glue.
• Familiarity with data product management tooling and data marketplace patterns.
• Previous experience in a technology modernisation or platform migration programme within government.
• Understanding of UK public sector data domains, including law enforcement, border control, or central government.