Lead PySpark Engineer
- Hiring Organisation
- Randstad Technologies
- Location
- London, South East, England, United Kingdom
- Employment Type
- Contractor
- Contract Rate
- £281 - £292 per day
Tier-1 financial services environment. Core Responsibilities Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark … unit testing, and data reconciliation frameworks to ensure "penny-perfect" accuracy. Technical Stack Engine: PySpark (Expert), Python (Clean Code/SOLID principles). AWS: EMR, Glue, S3, Athena, IAM, Lambda. Data Modeling: SCD Type 2, Fact/Dimension tables, Data Vault/Star Schema. Legacy: Proficiency in reading/ ...