Lead Test Engineer (SC Cleared)

About Scrumconnect

Scrumconnect Consulting is a UK-based digital transformation consultancy delivering agile, secure technology solutions for public and private sector clients. This is a fully remote role based in India, supporting our UK operations and client base. You will work closely with our UK leadership team and must be comfortable operating within UK regulatory and legal frameworks.

About the Role

We are looking for a Test Engineer to join a large-scale cloud data engineering programme, operating across a modern AWS-native technology stack including Apache Airflow, Amazon Athena, AWS Glue, S3, EMR, and DynamoDB.

You will own testing across automated pipelines, data workflows, and cloud infrastructure - identifying risks, championing test frameworks, and coaching colleagues in quality engineering practices.

This is a hands-on, technically deep role. You will write code, build and extend test frameworks, and take accountability for test quality across releases, while actively mentoring junior team members.

Key Responsibilities

Automated Testing

• Design, implement, and maintain automated test suites across data pipelines, cloud services, and applications

• Automate data and application testing tasks and build test coverage using existing or new infrastructure

Risk Identification & Reporting

• Identify and raise awareness of risks arising from automated test results

• Analyse and report on test activities, results, issues, and risks

• Translate technical findings into clear, prioritised communication

Framework Development

• Identify, evaluate, and implement new test frameworks

• Enhance existing frameworks to improve testing confidence and coverage

Pipeline & Data Testing

• Validate data pipelines using Apache Airflow, AWS Glue, Athena, and EMR

• Ensure data integrity, transformation accuracy, and performance under load

• Analyse data in multiple formats to validate new functionality

• Perform production data analysis to identify root causes of issues

Mentoring & Coaching

• Mentor and advise team members on testing best practices

• Contribute to a quality-first engineering culture

CI/CD & DevOps

• Integrate and manage automated tests within GitLab CI/CD pipelines

• Ensure automated execution and fast feedback loops

Technology Stack

Apache Airflow, Amazon Athena, AWS S3, AWS Glue, AWS EMR, AWS EC2, AWS ECR, AWS DynamoDB, AWS CloudWatch, AWS IAM, Python, Java, SQL, Bash, GitLab CI/CD, Jupyter Notebooks, Apache Spark, Terraform, Docker

Key Skills

• Strong proficiency in Python, Java, SQL, and Scripting (eg, Bash)

• Experience with Apache Airflow for orchestration and log analysis

• Hands-on experience with AWS services (S3, Glue, Athena, EMR, CloudWatch, IAM, DynamoDB, EC2, ECR)

• Experience using GitLab and GitLab CI/CD pipelines

• Ability to analyse data using Jupyter Notebooks and Amazon Athena

• Understanding of Apache Spark and EMR (basic to intermediate)

• Experience with infrastructure as code (Terraform)

• Familiarity with Docker and containerization

• Knowledge of Server Side and client-side encryption

• Understanding of dimensional data models and slowly changing dimensions

• Experience in data creation/generation for testing

• Strong analytical skills for validating data and identifying issues

• Ability to understand how customer requirements translate into functional solutions

Skills & Experience Required

• Proven experience as a Test Engineer or QA Engineer in cloud/data environments

• Strong Python skills for automation, data validation, and test development

• Experience testing data pipelines (data quality, transformation, performance)

• Experience designing or extending automated test frameworks

• Strong problem-solving and root cause analysis skills

• Ability to clearly communicate risks to technical and non-technical stakeholders

• Experience mentoring or coaching junior engineers

• SFIA Level 4 capability - works autonomously and influences others

• Active SC clearance (mandatory)

Diversity and Inclusion

At Scrumconnect Consulting, we believe that diversity drives innovation. We are committed to creating an inclusive environment where every individual is respected, valued, and supported. We welcome applications from candidates of all backgrounds and experiences.