Data Engineer

Data Engineer

Inside IR35

£450 - £480p/d

6months+

Fully Remote

SC Clearance Required

Data Engineer/python expert/scrum master required with SC clearance to work on a UK government account.

Must be UK based and hold a valid and current passport.

Technology requirements:

• Python 3 / Pyspark 3/4
• Python behave - for Behaviour Driven Development and testing
• Python coverage - Code coverage
• Strong knowledge of Databricks, including delta parquet data format and the medallion data architecture
• Strong knowledge of YAML.
• Understanding of Azure Devops
• Understanding of Git and code release best practices

The role will be to design and build core python libraries for ingesting various data sets. The overarching concept of this solution is that the libraries will utilise YAML files to control the transformation of all data sets (there will be exceptions, but this is the core ideology for this solution). Therefore, for each different data set the engineers will be required to create appropriate YAML files and utilise the libraries that will be created here without specific python programming.

As well as strong Python and programming experience, the resource will also need to create the YAML files for converting the incoming data in to the appropriate Bronze/Silver/Gold medallion architecture and create behaviour driven test using Python Behave.

All development will be released through Azure Devops pipelines, that will be created by a separate team, but an understanding of the technology and knowledge of Git best practices would be an advantage