Senior Data Engineer
Senior Data Engineer - Contract - London - 3 days on site - £520-550pd via Umbrella
Contract until Feb 27
Key Responsibilities
ETL Pipeline Architecture: Design, build, and maintain scalable ETL pipelines using Apache Airflow, Python, PySpark, and AWS Glue to transform complex financial datasets into actionable insights.
Data Integration: Lead the integration of diverse data sources, ensuring data consistency, accuracy, and timely delivery to downstream systems.
Containerization & Deployment: Utilize Docker for containerization and manage deployments within an AWS cloud environment (utilizing EKS or ECS).
Technical Leadership: As a Principal Engineer, provide technical guidance, code reviews, and mentorship to the engineering team, ensuring best practices in coding, architecture, and documentation.
Optimization: Drive performance tuning and continuous improvement of data ingestion and transformation processes.
Required Skills
ETL Development: Extensive experience building, testing, and deploying production-grade ETL pipelines in a cloud environment.
Python: Expert-level proficiency in Python for data processing, automation, and Scripting.
PySpark: Strong hands-on experience with PySpark for large-scale data transformation and processing.
AWS Data Stack: Deep hands-on experience with AWS Glue and related AWS services (S3, Lambda, Athena, Glue Catalog).
Orchestration: Proven experience with Apache Airflow for scheduling, monitoring, and managing complex workflows.
Docker: Proficiency in containerization using Docker and experience with AWS container services.
Nice-to-Have Skills
Capital Markets Experience: Prior experience in the financial sector, particularly in Pricing, Market Data, or Trade Processing.
Domain Knowledge: Familiarity integrating complex financial data sources, specifically regarding Discounting data (eg, BroCalc), DSO, Energy & Commodities data (FACTS), and Cash/Credit product data.
UI Awareness: Familiarity with modern UI frameworks (React, Next.js, TypeScript) to facilitate better collaboration with Front End delivery teams.