(PYSPARK+ FABRIC) Developer

Role: Developer (PySpark+ Fabric)

Location: London

Contract (6months +)

Hybrid (Inside IR35)

The Role

The role will be integral to realizing the customer's vision and strategy in transforming some of their critical application and data engineering components. As a global financial markets infrastructure and data provider, the customer keeps abreast of the latest cutting technologies enabling their core services and business requirements. The role is critical in this endeavour by the means of enabling the technical thought leadership and excellence required for the purpose.

Your responsibilities:

• Design, build, and optimise scalable data pipelines for batch and streaming workloads
• Develop and manage dataflows, and semantic models to support specific analytics related business requirements
• Implement complex transformations, aggregations, and joins ensuring performance, and reliability
• Implement and apply robust data validations, cleansing, and profiling techniques to ensure data accuracy and consistency across datasets
• Implement role-based access, data masking, and compliance protocols
• Performance tune and optimise jobs and workloads to reduce latency
• Work collaboratively with analysts and business stakeholders to translate requirements into technical solutions
• Create, maintain, and update documentation and internal knowledge repository

Your Profile

Essential skills/knowledge/experience:

• Ability to write Spark code for large scale data processing, including RDDs, DataFrames, and Spark SQL
• Hands-on experience with lakehouses, dataflows, pipelines, and semantic models
• Ability to build ETL workflows
• Familiarity with time-series data, market feeds, transactional records, and risk metrics
• Familiarity with Git, DevOps pipelines, and automated deployment
• Strong communication skills with a collaborative mindset to work with and manage stakeholders