Data Engineer
Data Engineer £53,000 per annum Bedford Permanent Hybrid Working Role Overview Join us at the Money and Pensions Service (MaPS) as a Data Engineer where you will design and implement data pipelines to connect operational systems, engineering data for analytics and business intelligence (BI) systems in a cloud-only environment, predominantly Microsoft Azure.This role offers the opportunity to work with Senior Engineers on cloud based data engineering pipelines and be part of developing our technical data architecture and platform.The Data Engineer reports to the Senior Data Engineer. The role is aligned to the Government Digital and Data Profession Capability Framework: Senior Data Engineer: https://ddat-capability-framework.service.gov.uk/role/data-engineer#data-engineerYou’ll be part of a dynamic team dedicated to helping people across the UK improve their financial wellbeing.Key Responsibilities: Data Pipelines and ETL Processes• ETL Implementation: Implement ETL processes within MaPS architectural patterns to extract, transform, and load data from various source systems into our reporting solutions.• Pipeline Development: Develop and configure meta-data driven data pipelines using data orchestration tools such as Azure Data factory and engineering tools like Apache Spark to ensure seamless data flow.• Monitoring and Failure Recovery: Implement monitoring procedures to detect failures or unusual data profiles and establish recovery processes to maintain data integrity.Azure Services• Utilise Cloud based data technology platforms, Data Lake based storage & Cloud data pipeline/orchestration tools for efficient data storage, processing, and transfer.• Ensure the scalability and reliability of data solutions using Azure's suite of services.Collaboration• Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet their needs.• Working within Agile methodology, helping to build clear user stories and work items.Data Governance and Quality• Ensure data governance policies are adhered to and maintain high standards of data quality and security.• Develop pipelines using best practices for data management and integration.• Ensure only thoroughly tested, high quality code and resulting Data Products are delivered to the MaPS business.Data...