Data Engineer (SC Cleared)
- Hiring Organisation
- scrumconnect ltd
- Location
- City, Newcastle Upon Tyne, United Kingdom
- Employment Type
- Permanent
- Salary
- GBP 65,000 - 75,000 Annual
data processing workflows. Root cause analysis Perform data analysis to identify and resolve root causes of pipeline failures and data quality issues - including reviewing EMR output logs and CloudWatch metrics. Data modelling Apply understanding of dimensional data models and slowly changing dimensions (SCD) to design and maintain well-structured … development and data processing SQL - used for querying, transformation, and validation across data stores PySpark - for distributed data processing using Apache Spark on AWS EMR Familiarity with basic data structures for constructing robust, scalable solutions Data processing & orchestration Apache Spark - understanding of distributed data processing architecture and execution Apache ...