Data Infrastructure Engineer
CACI's Consumer & Market Intelligence and Experience & Transformation groups provide data, software and consulting services to improve our clients' sales and marketing programmes.
We help clients find, retain and grow profitable customers through our understanding of consumer characteristics and behaviour. And we help commercial property developers and retailers plan retail provision in the UK and overseas.
C&MI and E&T Groups help clients shape the vision, define the strategy and deliver a truly integrated customer experience. Our services are based on in-depth understanding of individual consumer behaviour and marketing technology which influence the way consumers buy products and services through a combination of data, technology and consultancy.
About Data Platform:
The Data Platform will be built and managed "as a Product" to support a Data Mesh organisation.
The Data Platform focusses on enabling decentralised management, processing, analysis and delivery of data, while enforcing corporate wide federated governance on data, and project environments across business domains.
The goal is to empower multiple teams to create and manage high integrity data and data products that are analytics and AI ready, and consumed internally and externally.
What does a Data Infrastructure Engineer do?
A Data Infrastructure Engineer will be responsible to develop, maintain and monitor the data platform infrastructure and operations. The infrastructure and pipelines you build will support data processing, data analytics, data science and data management across the CACI business.
The data platform infrastructure will conform to a zero trust, least privilege architecture, with a strict adherence to data and infrastructure governance and control in a multi-account, multi-region AWS environment.
You will use Infrastructure as Code and CI/CD to continuously improve, evolve and repair the platform.
You will be able to design architectures and create re-useable solutions to reflect the business needs.
Responsibilities will include:
Collaborating across CACI departments to develop and maintain the data platform
Building infrastructure and data architectures in Cloud Formation, and SAM.
Designing and implementing data processing environments and integrations using AWS PaaS such as Glue, EMR, Sagemaker, Redshift, Aurora and Snowflake
Building data processing and analytics pipelines as code, using Python, SQL, PySpark, spark, CloudFormation, lambda, step functions, Apache Airflow
Monitoring and reporting on the data platform performance, usage and security
Designing and applying security and access control architectures to secure sensitive data
You will have:
6+ years of experience in a Data Engineering role.
Strong experience and knowledge of data architectures implemented in AWS using native AWS services such as S3, DataZone, Glue, EMR, Sagemaker, Aurora and Redshift.
Experience administrating databases and data platforms
Good coding discipline in terms of style, structure, versioning, documentation and unit tests
Strong proficiency in Cloud Formation, Python and SQL
Knowledge and experience of relational databases such as Postgres, Redshift
Experience using Git for code versioning, and lifecycle management
Experience operating to Agile principles and ceremonies
Hands-on experience with CI/CD tools such as GitLab
Strong problem-solving skills and ability to work independently or in a team environment.
Excellent communication and collaboration skills.
A keen eye for detail, and a passion for accuracy and correctness in numbers
Whilst not essential, the following skills would also be useful:
Experience using Jira, or other agile project management and issue tracking software
Experience with Snowflake
Experience with Spatial Data Processing