Data Engineer
We are looking for an experienced Data Engineer to support a data migration and integration programme within a relatively low maturity Azure environment.
The role will focus on designing and building robust data pipelines, migrating data from a wide range of legacy and enterprise source systems, and establishing scalable data flows into Azure Data Lake and Databricks. A key part of the role will also involve integrating data into a proprietary data transformation solution where applicable.
Key Skills & Experience
- Azure Data Engineering
- Azure Data Factory (ADF)
- Azure Data Lake
- Databricks
- Building ETL / ELT pipelines
- Large-scale data migration and integration
- Oracle and/or Oracle Fusion data sources
- SharePoint data extraction and integration
- Working with multiple and varied source systems
- Data transformation, mapping and validation
- Designing data pipelines in environments where architecture and processes are still developing
Key Responsibilities
- Design, build and maintain data migration and ingestion pipelines within Azure.
- Support the setup and development of the Azure Data Lake environment.
- Build integrations between source systems, Azure Data Lake and Databricks.
- Develop and manage pipelines using Azure Data Factory (ADF).
- Extract and migrate data from a variety of source systems, including:
- Oracle
- Oracle Fusion
- Informatica
- SharePoint, which will be an initial priority
- Additional enterprise and legacy systems as the programme develops
- Help define scalable data ingestion, transformation and migration patterns for future source systems.
- Support data quality, validation and troubleshooting throughout the migration process.
- Work closely with technical and business stakeholders to understand source data, target structures and transformation requirements.
- Help improve the overall maturity of the Azure data environment, introducing appropriate engineering standards and best practices.