Data engineer (Pyspark)

Senior Data Engineer (Developer) – Pyspark

We are supporting a leading global financial markets infrastructure and data provider as they modernise and scale their core data engineering capabilities. This role sits at the centre of their transformation programme, delivering high-quality data pipelines, models, and platforms that underpin critical services across the business.

Key Responsibilities

Design, build, and optimise scalable data pipelines for both batch and streaming workloads:

• Develop dataflows and semantic models aligned to analytics and reporting needs
• Implement complex transformations and performance-focused data processing logic
• Apply data validation, cleansing, and profiling techniques to ensure accuracy and consistency
• Implement access controls, data masking, and compliance-aligned security protocols
• Tune workloads and optimise performance across Spark, Fabric, and Azure components
• Translate business requirements into technical solutions through close collaboration with analysts and stakeholders
• Maintain clear documentation and contribute to internal knowledge repositories

Essential Skills

Strong experience developing within Microsoft Azure and Microsoft Fabric:

• Proficiency in Spark programming including DataFrames, RDDs, and Spark SQL
• Python and PySpark development experience, including notebook-based workflows
• Hands-on experience with Spark streaming and batch processing
• Delta table optimisation and Fabric Spark job development
• Solid Java programming and OOP understanding
• Experience working with relational and NoSQL databases
• Familiarity with GitLab, unit testing, and CI/CD pipelines
• Strong troubleshooting ability and experience working in Agile environments
• Excellent communication skills with stakeholder-facing experience
• Practical experience building ETL workflows, lakehouse architectures, dataflows, and semantic models
• Exposure to time-series data, financial market feeds, transactional records, and risk-related datasets