Data Engineer - Luton (hybrid working) - £45k + Benefits - GCP

Data Engineer - Luton (hybrid working) - £45k + Benefits

We are seeking a skilled and experienced Data Engineer to join our growing Data & Analytics team. As a Data Engineer, you will be responsible for designing, building, and maintaining our data infrastructure on Google Cloud Platform (GCP). Reporting to our Data Scientist, your primary focus will be on building robust and scalable data pipelines to ingest, process, and transform data from various sources, utilizing Cloud Composer for orchestration and the broader Google suite for data management. You will play a crucial role in ensuring the availability, reliability, and quality of data for analytics and reporting across the organisation.

Key Responsibilities:

Data Pipeline Development & Management:

• Design, develop, and maintain scalable and reliable data pipelines using Cloud Composer (Apache Airflow) on GCP, adhering to infrastructure-as-code principles.
• Implement data ingestion processes from diverse sources, including databases, APIs, flat files (eg, CSV exports from vineyard management systems), and streaming data (eg, IoT sensor data from vineyards).
• Develop and maintain data models, ETL (Extract, Transform, Load) processes, and data warehousing solutions in BigQuery, considering both batch and streaming data ingestion patterns.
• Automate data quality checks and monitoring using tools within the Google Cloud ecosystem to ensure data accuracy and consistency, with alerts for anomalies.
• Optimize data pipelines for performance and cost-efficiency, considering factors like data volume, processing complexity, and query patterns.
• Implement data loss prevention (DLP) measures to identify and de-identify Personally Identifiable Information (PII).

GCP Infrastructure Management:

• Manage and maintain GCP services, including Cloud Storage, BigQuery, Cloud Functions, Dataflow, Pub/Sub, and other relevant services, following best practices for security and cost optimization.
• Implement infrastructure-as-code (IaC) using tools like Terraform to automate infrastructure provisioning and management, ensuring consistency and repeatability.
• Monitor system performance and troubleshoot issues related to data pipelines and GCP infrastructure, including setting up alerts for key metrics.
• Ensure the security and compliance of data and infrastructure on GCP, implementing IAM (Identity and Access Management) controls and encryption at rest and in transit.

Data Governance & Quality:

• Collaborate with data analysts and business stakeholders to understand data requirements and define data quality standards, particularly related to wine production, sales, and customer data.
• Implement data governance policies and procedures to ensure data integrity and compliance with relevant regulations (eg, GDPR).
• Develop and maintain data catalogs and data lineage documentation to ensure transparency and discoverability of data assets.

Collaboration & Communication:

• Work closely with data scientists, data analysts, and business stakeholders to understand data needs and provide data solutions, including feature engineering for machine learning models.
• Communicate technical concepts effectively to both technical and nontechnical audiences.
• Participate in code reviews and knowledge sharing within the team.
• Stay up-to-date with the latest GCP technologies and best practices, and proactively recommend improvements to the data infrastructure.

Required Skills & Experience:

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 3+ years of experience in data engineering, with a focus on building and maintaining data pipelines.
• Strong proficiency in Python and SQL.
• Extensive experience with Google Cloud Platform (GCP) services, including:
• Cloud Composer (Apache Airflow)
• BigQuery
• Cloud Storage
• Dataflow
• Pub/Sub
• Cloud Functions
• IAM
• Experience with data modelling, ETL processes, and data warehousing concepts.
• Experience with infrastructure-as-code (IaC) tools like Terraform.
• Solid understanding of data governance principles and data quality management.
• Experience working with version control systems (eg, Git).
• Excellent problem-solving and analytical skills.
• Strong communication and collaboration skills.

This is a great chance to join an exciting company going through a period of growth that can offer stability and growth. For more information and a full job description, please contact Ben Turner at BRT Consulting.