Azure Databricks Architect
Role Overview
We are seeking an experienced Senior Data Architect to lead the design, assessment, and implementation of enterprise-scale data platforms using Azure Databricks and Microsoft Fabric. The successful candidate will have extensive experience in modern lakehouse architectures, data governance, enterprise data integration, and stakeholder engagement. This role will be responsible for reviewing existing implementations, defining target-state architectures, and delivering scalable, secure, and governed data solutions.
Key Responsibilities
- Design and implement modern lakehouse architectures using Azure Databricks and Delta Lake.
- Define and establish Medallion Architecture (Bronze, Silver, Gold) data layers to support analytics, reporting, and operational use cases.
- Configure and manage Unity Catalog, including metastore setup, catalog and schema design, governance policies, and access controls.
- Architect and manage Azure Data Lake Storage Gen2 environments, including storage containers, security models, lifecycle management, and integration with Databricks.
- Implement and optimize Microsoft Fabric integrations, including Databricks Unity Catalog mirroring and Power BI connectivity through Databricks SQL Warehouses.
- Design and oversee ingestion frameworks for enterprise data sources, including:
- Microsoft SQL Server (on-premises and Azure SQL)
- Oracle Databases
- Develop and optimize Apache Spark workloads using PySpark or Scala, ensuring performance, scalability, and operational efficiency.
- Assess existing or partially completed data platform implementations and provide remediation, optimization, and modernization recommendations.
- Collaborate with enterprise architects, data engineers, business stakeholders, and leadership teams to align data architecture with business objectives.
- Present architectural assessments, recommendations, and technical roadmaps to senior stakeholders in a clear and business-focused manner.
Required Skills & ExperienceEssential Requirements
- 8+ years of experience in Data Architecture, including at least 3 years of hands-on Azure Databricks implementation in production environments.
- Proven expertise in designing and implementing Medallion Architecture (Bronze, Silver, Gold) using Delta Lake on Azure Databricks.
- Strong hands-on experience with Unity Catalog, including:
- Metastore configuration
- Catalog and schema design
- Data governance
- Role-based access controls
- Experience configuring and managing Azure Data Lake Storage Gen2, including:
- Storage containers
- Access Control Lists (ACLs)
- Lifecycle management policies
- Databricks integration
- Hands-on experience with Microsoft Fabric, particularly:
- Mirrored Azure Databricks Unity Catalog
- Power BI integration with Databricks SQL Warehouses
- Experience integrating enterprise data sources such as:
- Microsoft SQL Server (on-premises and Azure SQL)
- Oracle Databases
- Strong Apache Spark expertise using PySpark and/or Scala, including performance tuning, partitioning strategies, and workload orchestration.
- Experience evaluating and remediating incomplete, underperforming, or legacy data platform implementations.
- Strong stakeholder management and communication skills with experience engaging senior leadership, including CIO, CDO, and Architecture Review Boards.