Software Developer - Security Clearance
Job Title: Software Developer (Python/Data/AI Automation)
Clearance: Active SC is required for this role - client has the ability to transfer
Contract: 6 months
Rate: £475pd inside IR35
Location: Predominantly remote with occasional travel to Newcastle (expenses covered)
Overview:
We're supporting a major UK central government transformation programme focused on modernising and centralising how financial support services are delivered across the UK.
This is an Alpha phase programme, meaning engineers will be involved from the ground up - helping shape architecture, build prototypes, solve complex data challenges and define the future state of a large-scale national platform.
This role sits at the intersection of software engineering, data processing, automation and cloud engineering, with a strong focus on building scalable solutions across large datasets and modern AWS infrastructure.
The Role:
You'll work closely with architects, engineers and wider technical teams to design, build and troubleshoot scalable data-driven applications.
This role requires someone who is comfortable working across Python development, data engineering, cloud infrastructure, automation and platform tooling, while helping translate business/customer requirements into working technical solutions.
Key Responsibilities:
- Build and maintain scalable applications using Python, SQL and PySpark
- Develop and support large-scale data processing pipelines using Apache Spark
- Use Apache Airflow for workflow orchestration and task scheduling
- Work with AWS EMR environments and review logs/output to troubleshoot issues
- Query, validate and analyse datasets using Jupyter Notebooks and/or Amazon Athena
- Perform root cause analysis across data pipelines and platform issues
- Design and support dimensional data models including slowly changing dimensions and historical data capture
- Build and maintain cloud-native solutions across AWS services
- Support containerised application development using Docker
- Manage infrastructure through Terraform (IaC)
- Support CI/CD pipelines, deployments and release management using GitLab
- Use GitLab tags for version control and release deployments
- Work with encryption/security best practices including Server Side and client-side encryption
- Collaborate with stakeholders to translate customer requirements into technical functionality
- Follow engineering best practices across delivery team
Technical Environment:
Programming/Data:
- Python
- SQL
- PySpark
- Apache Spark
- Apache Airflow
AWS/Cloud:
- AWS EMR
- S3
- Glue
- CloudWatch
- IAM
- EC2
- ECR
- DynamoDB
- Lake Formation
- Amazon Athena
AI/Intelligent Document Processing:
- Amazon Textract
- Amazon Comprehend
DevOps/Infrastructure:
- Docker
- Terraform
- GitLab CI/CD
- GitLab release tagging
Skills & Experience Required:
- Strong hands-on Python development experience
- Strong experience building data processing pipelines
- Experience working with large datasets and distributed processing frameworks
- Strong AWS cloud experience
- Experience troubleshooting data issues and identifying root causes
- Understanding of data modelling principles
- Experience with CI/CD pipelines and deployment processes
- Experience working in Agile delivery teams
- Ability to work in fast-moving discovery/Alpha environments
If you are available and interested in this opportunity, please apply for further information. Please note that due to high volumes of applications we are unable to contact every applicant. If you do not hear back from us within 7 days of sending your application, please assume that you have not been successful on this occasion.
At Lucid, we celebrate difference and value diverse perspectives, underpinned by our values 'Honesty, Integrity and Pragmatism'. We are proud to provide equal opportunities in line with our Diversity and Inclusion policy and welcome applications from all suitably qualified or experienced people, regardless of personal characteristics. If you have a disability or health condition and seek support throughout the recruitment process, please do not hesitate to contact us via the details below.