Senior Data/Platform Engineer

Job Description:

Duties and Responsibilities

• Lead solution design activities, collaborating with peers and mentoring junior colleagues to define and execute the team backlog.
• Develop, test, and document scalable ETL/ELT data pipelines and workflows using Databricks and Azure Synapse to ingest and transform data from a variety of sources.
• Administer and maintain Azure data platform components including Synapse, Databricks, ADLS Gen2, Key Vault, networking (VNets, NSGs, Managed Private Endpoints) and access control (RBAC, ACLs).
• Manage infrastructure-as-code across Dev, Staging, and Production environments using Pulumi (and equivalents such as Terraform/Bicep).
• Design and operate CI/CD pipelines using GitHub Actions (with OIDC federation) and/or Azure DevOps, supporting trunk-based development practices.
• Administer Databricks workspaces - cluster policies, Secret Scopes, Repos/Git integration, Workflow job health, and Unity Catalog governance.
• Monitor platform and pipeline health using Azure Monitor, Log Analytics, KQL, and Azure Dashboards; triage and resolve incidents.
• Implement robust data security and ensure compliance with data privacy regulations; manage service principals, Managed Identities, and least-privilege access.
• Carry out routine platform operations: patching, backups, storage life cycle, tagging, access reviews, DR readiness, and runbook execution.
• Identify and address performance bottlenecks and data quality issues to ensure data accuracy and reliability.
• Work with testers to ensure automated test plans are in place and agree test packs for UAT; review peers' work and take accountability for the quality of squad deliverables.
• Collaborate with stakeholders and analysts to understand data requirements and deliver clean, reliable, accessible data.
• Ensure solution designs align with the Allianz Data Technology Strategy, maintaining and enhancing Allianz standards and Service Excellence.
• Maintain technical documentation to Allianz standards (eg, Grimlock) and stay current with industry trends, driving continuous improvement and innovation.

Must Have

• Databricks: hands-on experience building and optimizing pipelines, managing Delta Lake, and administering workspaces (cluster policies, Unity Catalog, Secret Scopes, Workflows).
• Python/PySpark: strong programming skills for data processing, automation, and Scripting.
• Azure data stack: Synapse, Databricks, ADLS Gen2, Key Vault - including Linked Services, Managed Identity, and Spark Pool configuration.
• Azure platform fundamentals: compute, storage, networking (VNets, NSGs, Private Endpoints), identity and RBAC.
• CI/CD: GitHub Actions (with OIDC federation) and/or Azure DevOps for data and platform deployments.
• Infrastructure-as-Code: Pulumi (or Terraform/Bicep) across multiple environments.
• Scripting: PowerShell and Bash for platform automation.
• Monitoring & observability: Azure Monitor, Log Analytics, KQL.
• Big data file formats: Parquet and Delta Lake.
• Cloud-native data modelling and ETL/ELT frameworks on Azure.