Data Engineer - ONS - SEO

You'll have a passion for combining data engineering principles with software development to create data products that enhance our ability to process, store, and retrieve information across various platforms. You will work closely with a multidisciplinary team of developers, engineers, and production staff, ensuring the smooth and efficient development and deployment of solutions.

Responsibilities 

• Design, develop, and maintain ETL pipelines and data processing workflows to manage and manipulate datasets, ensuring data quality, consistency, and accuracy.
• Create intuitive and user-friendly interfaces for internal production teams, making sure that all software products are highly usable and meet the users' requirements.
• Manage and enhance the Python codebase used for both data processing and user interface functionality, ensuring clean, maintainable, and well-documented code.
• Build and maintain scalable, secure, and efficient cloud-based solutions, including designing database schemas and managing data flows.
• Implement software development best practices, including version control (Git), continuous integration, testing, and deployment (CI/ CD).
• Troubleshoot and optimise existing data pipelines and software, ensuring performance improvements and resilience in production environments.
• Collaborate with data scientists, analysts, and other teams to ensure efficient data integration, storage, and processing, and the transfer of appropriate data engineering knowledge.
• Provide line management and support to junior team members, guiding their growth and development.
  Person specification

  Essential Skills Criteria:

  • Ability to determine customer requirements and translate these into delivery design for data processing.
  • Ability to design, write, and operate ETL pipelines, in the context of distributed processing, applying coding, data, and documentation standards, in the language required by the business area.
  • Experience of developing in one of the data platforms ONS use; Cloudera or GCP. With experience in Python and knowledge of SQL and/ or R would be an advantage.
  • Experience of writing clean, efficient and well documented code for data processing tasks to a specification.
  • Experience of using and applying code version control across all work using Git with GitHub or GitLab.
  • Ability to lead the development of ETL pipelines, including effective delegation of work.
  • Good understanding of the principles of data processing in a distributed system, and ability to use this understanding to ensure and design robust coding in a distributed environment, including the creation of flow charts to illustrate coding flows and unit tests and integration testing.
    Behaviours

    We'll assess you against these behaviours during the selection process:

    • Leadership
    • Changing and Improving
    • Managing a Quality Service
    • Working Together
      Technical skills

      We'll assess you against these technical skills during the selection process:

      • Programming and Build
      • Data Integration Design