PySpark Developer (Microsoft Fabric) - 6-Month Contract - London
PySpark Developer (Microsoft Fabric) - 6-Month Contract - London
Contracting role based in London (Hybrid, 1-3 days in office) for a 6-month engagement. You will design, build, and maintain data pipelines powering an end-to-end mastering flow for a leading global financial markets infrastructure and data provider, working across PySpark, Azure Cosmos DB, and Microsoft Fabric.
Key Responsibilities
- Build and maintain PySpark pipelines for ingestion, normalisation, bi-temporal processing, and publication of financial instrument data
- Develop and optimise Azure Cosmos DB read/write patterns and integrate with external APIs for Minting & Matching
- Implement data quality checks, comprehensive unit/integration tests, and CI/CD pipeline management via GitLab CI
- Monitor and troubleshoot production pipelines using Eventstream telemetry, KQL, and DataDog
- Translate business requirements into technical solutions, maintaining documentation and knowledge repositories
What You Will Ideally Bring
- Strong proficiency in Python and PySpark, with hands-on experience building streaming mastering pipelines
- Solid Azure experience - Cosmos DB (NoSQL), ADLS/OneLake, and Key Vault
- Strong understanding of temporal data modelling, data quality frameworks, and CI/CD practices
- Experience with lakehouses, dataflows, and semantic models within Microsoft Fabric - preferred
- Domain knowledge of financial instruments and reference data - advantageous
Contract Details
- Duration: 6 months (with potential extension)
- Day Rate: Up to £450 per day (Inside IR35)
- Location: - Hybrid London
- Start Date: ASAP