Senior Data Engineer
The Role
The Interconnectors Data Platform (ICDP) is a cloud-based data warehouse enabling critical commercial, financial modelling, and operational decision-making. As the platform moves toward a modernized Medallion Architecture and Azure-native ingestion patterns, we are hiring a Senior Data Engineer to provide architectural direction and hands-on technical leadership.
Your responsibilities:
Data Architecture & Platform Engineering
- Lead the design and implementation of scalable data architectures, including Bronze/Silver/Gold layered models.
- Drive the platform’s architectural roadmap and ensure alignment with modern engineering best practices.
- Develop robust, secure, and observable ingestion and transformation pipelines.
Pipeline Development & Operations
- Lead the migration from legacy ETL tools (e.g., Matillion) into modern Azure-based pipelines, including Azure Functions, Azure Data Factory (ADF), and event-driven frameworks.
- Build and maintain performant SQL-based transformations, curated layers, and reusable data models.
- Ensure CI/CD, testing, version control, and observability are fully embedded in workflows.
Data Quality & Governance
- Implement data validation, reconciliation, profiling, and auditability across platform layers.
- Work with business stakeholders to ensure output meets analytical and operational needs.
Leadership
- Mentor data engineers and contribute to the growth of technical capability within the ICDP team.
- Collaborate with Product teams, IT&D, and external partners to deliver high-quality outcomes.
- Serve as a technical authority for engineering approaches, patterns, and standards.
Your Profile
- Python: Strong, hands-on experience building production-grade data pipelines, transformations, and orchestration using Python.
- Advanced SQL: Expert-level analytical SQL, query optimization, and data modelling.
- Azure Cloud: Experience with Azure Functions, Azure Data Factory, Azure Storage, Key Vault, and cloud security fundamentals.
- Data Warehousing: Deep understanding of data architecture principles, dimensional modelling, and scalable enterprise data design.
- Strong grounding in Git, CI/CD, automated testing, branching strategies, and modern engineering practices.
- Experience with API ingestion, SFTP ingestion, and resilient pipeline design.
Soft Skills
- Strong problem-solving and architectural thinking.
- Excellent communication and stakeholder collaboration skills.
- Ability to provide clarity and leadership in complex or ambiguous technical landscapes.
Desirable skills/knowledge/experience:
Experience contributing to or leading data-platform re-architecture programs.
- Exposure to Medallion/Lakehouse patterns or Databricks-style ecosystems.
- Experience in regulated or high-assurance data environments.
- Collaboration with modelling teams or environments where engineering and modelling intersect.