Lead PySpark Engineer
- Hiring Organisation
- Randstad Technologies Recruitment
- Location
- City of London, London, United Kingdom
- Employment Type
- Contract
- Contract Rate
- £281 - £292/day 388
PySpark Engineer Lead As the Technical Lead, you will drive the high-stakes migration of legacy SAS analytics to a modern, cloud-native PySpark ecosystem on AWS. This isn't just a lift and shift you will refactor complex procedural logic into scalable, production-ready distributed pipelines … Engineering Leadership: Design and develop complex ETL/ELT pipelines and Data Marts using PySpark, EMR, and Glue. Legacy Modernisation: Architect the conversion of SAS Base/Macros into modular, testable Python code using SAS2PY and manual refactoring. Performance Tuning: Optimise Spark execution (partitioning, shuffling, caching) to ensure cost ...