Senior Technical Lead

Job Title: Senior Technical Lead
Location: Norwich, Norfolk (3 Days a week)
Job Type: Contract (Inside IR35)
Duration: 6 Months The Role

We are looking for a Senior Technical Lead who combines hands-on engineering excellence with strong leadership and stakeholder management. You will own the end-to-end technical delivery of data platforms and pipelines built in AWS-with a focus on AWS Glue, Managed Workflows for Apache Airflow (MWAA), and Python-and collaborate closely with Directors, Senior Architects, and Program Leadership to deliver business outcomes at scale.

This is a player-coach role: you will design, build, review, and optimize complex data workflows while mentoring engineers and driving engineering best practices.

Ideal for: Someone who has delivered multiple production programs in a modern AWS data engineering landscape, can communicate trade-offs clearly to senior stakeholders, and can lead teams through ambiguity to predictable, high-quality outcomes.

Your Responsibilities:

• Lead the design and implementation of scalable, secure, and cost-efficient ETL/ELT pipelines using AWS Glue, Python (PySpark), and MWAA (Airflow).
• Define solution architectures, data models, orchestration patterns, and CI/CD for data workflows.
• Own the technical roadmap, decomposition, and delivery plan-including sizing, sprint planning, and risk mitigation.
• Drive performance optimization (e.g., partitioning strategies, Glue job tuning, job bookmarks, dynamic frames vs DataFrames, retry/backoff strategies in Airflow).
• Ensure robust observability (logging, metrics, tracing) and data quality (unit tests, Great Expectations/Deequ-style checks, validations).
• Act as the technical point of contact for Senior Architects, and Program Managers; translate business needs into technical designs and delivery milestones.
• Present architecture decisions, trade-offs, and TCO to senior stakeholders with clarity, data, and rationale.
• Manage vendor/partner coordination where relevant.
• Establish coding standards, code review practices, branching strategies, and secure-by-design principles.
• Implement DevSecOps for data: infrastructure-as-code (IaC), secrets management, environment promotion, and automated testing.
• Ensure compliance with data governance, security, and regulatory requirements (e.g., PII/PCI, encryption, auditability, lineage).
• Mentor and upskill engineers; foster a culture of learning, ownership, and continuous improvement.

Your Profile

Essential skills/knowledge/experience:

• 10+ years of total experience in software/data engineering, with 5+ years leading delivery of production solutions in an AWS data engineering environment.

Advanced hands-on expertise with:

• Python (including PySpark & data engineering patterns)
• AWS Glue (Jobs, Crawlers, Glue Studio, Glue Catalog, PySpark, Job bookmarks)
• MWAA (Apache Airflow) (DAG design, scheduling, sensors, retries, XComs, task isolation, best practices)

Strong across broader AWS services:

• S3, Lambda, Step Functions, IAM, CloudWatch, KMS, Secrets Manager, Athena, EMR (nice to have), Redshift (nice to have)
• Proven experience delivering multiple end-to-end programs (architecture build test deploy operate) with measurable outcomes (SLAs, cost targets, performance).
• Excellent stakeholder communication and executive presence; able to engage Directors, Senior Architects, and Program Leadership.
• Solid grounding in data modeling, data governance, security/compliance, and cost optimization on AWS.
• Experience with CI/CD (e.g., CodePipeline/GitHub Actions/Bitbucket Pipelines), IaC (CloudFormation/Terraform), and containerization (Docker).
• Architectural thinking: designs for scale, reliability, cost, and evolvability.
• Delivery excellence: breaks down complex work, sets milestones, manages risks, and delivers on time.
• Communication & influence: distills complexity for senior stakeholders; backs decisions with data.
• Hands-on leadership: sets the technical bar through reviews, pairing, and exemplars.
• Ownership & clarity: aligns teams on problem statements, success criteria, and measurable outcomes.

Languages:

• Python (PySpark), SQL

AWS:

• Glue, MWAA (Airflow), S3, IAM, KMS, CloudWatch, Lambda, Step Functions, Athena, Redshift (nice), EMR (nice)

DevOps:

• Git, CI/CD (CodePipeline/GitHub Actions), Terraform/CloudFormation, Docker

Data Quality/Observability:

• Great Expectations/Deequ (nice), OpenLineage (nice)

Desirable skills/knowledge/experience:

• Domain experience in BFSI (risk, pricing, regulatory reporting, underwriting, fraud, payments, or actuarial data).
• Experience with event-driven and near-real-time pipelines (Kafka/Kinesis, streaming ETL).
• Knowledge of data quality frameworks (Great Expectations, Deequ) and data lineage/catalog (Atlas, Alation, Collibra).
• Exposure to Databricks or EMR for advanced Spark workloads.
• Certifications: AWS Solutions Architect / Data Analytics / DevOps Engineer.
• Prior experience leading multi-team programs with offshore/nearshore models.

JBRP1_UKTJ