Data Engineer - SC Cleared

Data Engineer - SC Cleared - mostly remote - 12 months - £386

Required Skills & Qualifications

• Strong hands on experience with ETL development and orchestration (Informatica, Azure, or AWS).
• Solid AWS cloud experience, including working with core data services.
• Expertise in building distributed data pipelines using EMR, PySpark, or similar technologies.
• Strong data processing and transformation experience across large datasets.
• Proficiency in PySpark, Python, and SQL for data manipulation and automation.
• Understanding of data modelling, data warehousing concepts, and performance optimization.
• Familiarity with CI/CD tools (DevOps, GitHub, GitLab).
• Exposure to data governance, metadata management, and data quality frameworks.
• Experience working in Agile environments is a plus.

Key Responsibilities

• Develop, maintain, and optimize ETL pipelines using tools such as Informatica, Azure Data Factory, AWS Glue or Azure Data Factory
• Build and manage cloud based data pipelines leveraging AWS services (eg, EMR, S3, Lambda, Glue).
• Implement scalable data processing workflows using PySpark, Python, and SQL.
• Design and support data ingestion, transformation, and integration processes across structured and unstructured data sources.
• Collaborate with data architects, analysts, and business stakeholders to understand requirements and deliver reliable data solutions.
• Monitor pipeline performance, troubleshoot issues, and ensure data quality and reliability.
• Contribute to best practices for data engineering, including version control, CI/CD, and automation.