Developer (PySpark + Fabric)

If you require any support in completing your application or need this document in an alternative format, please contact the recruitment team at UKI.recruitment@tcs.com or call the TCS London Office on 020 3155 2100 . Please include the subject line: “Application Support Request” .

Role: Developer (PySpark + Fabric)

Job Type: Permanent

Location: London, UK

Mode of Working: Hybrid

Working Hours: 40 hours per week (Full-time)

Are you looking to build innovative data engineering solutions using PySpark, Microsoft Fabric, and Azure technologies?

We have an exciting opportunity for you - Developer (PySpark + Fabric)

Careers at TCS: It Means More

TCS is a purpose-led transformation company, built on belief. We go beyond helping businesses transform through technology—we enable them to make a meaningful difference to the people and communities they serve. Our clients include leading organisations across the UK and globally.

At TCS, it means more—more opportunity to make an impact that matters through challenging projects that demand innovation, collaboration, and thought leadership.

• Be part of a leading global financial markets infrastructure and data provider.
• Work within a team transforming and accelerating SDLC processes for critical application and data engineering platforms.
• Contribute to ambitious business transformation initiatives using modern cloud and data technologies.

The Role

As a Developer (PySpark + Fabric), you will play a key role in transforming critical data engineering capabilities within a leading financial services environment.

You will design, build, and maintain data pipelines that support the end-to-end mastering lifecycle, from data ingestion and normalisation through to bi-temporal processing, publication, and analytics consumption. Working closely with data architects, domain experts, quality engineers, and business stakeholders, you will deliver scalable, high-performance solutions using PySpark, Microsoft Fabric, and Azure technologies.

Key Responsibilities

• Design, develop, and maintain PySpark data pipelines to ingest, transform, and master financial instrument and reference data.
• Implement and optimise bi-temporal data processing solutions to support complex mastering and historical data requirements.
• Develop and optimise Azure Cosmos DB data models, partitioning strategies, indexing policies, and Change Feed integrations.
• Build and maintain integrations with external APIs and data enrichment services, implementing robust authentication, retry, and error-handling mechanisms.
• Develop publication pipelines to distribute mastered data through Microsoft Fabric and downstream platforms.
• Implement data quality controls and validation frameworks to ensure data accuracy, consistency, and reliability.
• Develop and maintain unit, integration, and end-to-end test suites using PyTest and related frameworks.
• Support CI/CD processes for application deployment, infrastructure provisioning, and automated testing.
• Define and maintain configuration-driven solutions using YAML and JSON-based frameworks.
• Monitor, troubleshoot, and optimise production pipelines using observability, telemetry, and logging tools.
• Develop dataflows, ETL processes, semantic models, and analytics-ready datasets to support reporting and business requirements.
• Implement data governance controls, including role-based access, data masking, and compliance requirements.
• Optimise workload performance, scalability, and resource utilisation across distributed processing environments.
• Collaborate with business stakeholders, analysts, and engineering teams to translate requirements into technical solutions.
• Create and maintain technical documentation, standards, and knowledge repositories.

Your Profile:

Essential skills/knowledge/experience

• Strong proficiency in Python, PySpark, Spark SQL, DataFrame APIs, Structured Streaming, and Java.
• Experience developing and maintaining large-scale data processing and mastering pipelines using PySpark.
• Strong experience working with complex and nested data structures in distributed data environments.
• Experience implementing API integrations, including secure HTTPS services, authentication, and retry mechanisms.
• Hands-on experience with Azure Cosmos DB, including data modelling, partitioning, indexing, throughput optimisation, and Change Feed processing.
• Experience with Azure Data Lake Storage (ADLS), OneLake, ABFS, and Azure Key Vault.
• Strong understanding of data modelling concepts, including temporal data patterns such as bi-temporal modelling, SCD Type 2, and effective dating.
• Experience with data quality and validation frameworks such as Great Expectations.
• Strong testing discipline using PyTest, test fixtures, mocking frameworks, and code coverage tools.
• Experience with Git-based CI/CD platforms such as GitLab CI, Azure DevOps, or similar.
• Experience working with YAML and JSON configuration files and Infrastructure-as-Code concepts.
• Strong experience with Spark for large-scale data processing using DataFrames, RDDs, and Spark SQL.
• Hands-on experience developing ETL workflows, lakehouse solutions, dataflows, pipelines, and semantic models.
• Familiarity with financial market data, time-series data, transactional datasets, or risk-related data domains.
• Strong communication and stakeholder engagement skills.

Desirable skills/knowledge/experience

• Experience with Microsoft Fabric, including Spark Job Definitions, Eventstream, Lakehouses, Data Pipelines, and Notebooks.
• Domain knowledge of financial instruments and reference data, including ISINs, CUSIPs, LEIs, PermIDs, and asset classifications.
• Experience with entity resolution, record matching platforms, and third-party enrichment services.
• Experience with Delta Lake and Change Data Feed (CDF) patterns.
• Knowledge of Cosmos DB SDK optimisation techniques, including bulk operations, optimistic concurrency, and RU optimisation.
• Experience using Jinja2 for configuration management or code generation.
• Familiarity with SonarQube for code quality and security analysis.
• Experience working within monorepo environments and Python package management frameworks.
• Experience preparing and optimising datasets for Power BI reporting and analytics.
• Knowledge of OneLake, Azure Data Lake, and distributed cloud data platforms.
• Understanding of regulatory and compliance requirements such as GDPR and SOX.

Rewards & Benefits

TCS is consistently voted a Top Employer in the UK and globally. Our competitive salary packages feature pension, health care, life assurance, laptop, phone, access to extensive training resources and discounts within the larger Tata network.

We offer health & wellness initiatives and sports events; we are the proud sponsor of the

London Marathon.

Diversity, Inclusion and Wellbeing

Tata Consultancy Services UK&I is committed to meeting the accessibility needs of all individuals in accordance with the UK Equality Act 2010 and the UK Human Rights Act 1998.

We welcome and embrace diversity in race, nationality, ethnicity, disability, neurodiversity, gender identity, age, physical ability, gender reassignment, sexual orientation. We are a disability inclusive employer and encourage disabled people to apply for this role.

As a Disability Confident Employer, we offer an interview to applicants with disabilities or long-term conditions who meet the minimum criteria for the role. Please email us at UKI.recruitment@tcs.com if you would like to opt in.

If you are an applicant who needs any adjustments to the application process or interview, please contact us at UKI.recruitment@tcs.com with the subject line: “Adjustment Request” or call TCS London Office 02031552100 / +44 204 520 2575 to request an adjustment. We welcome requests prior to you completing the application and at any stage of the recruitment process.

Next Steps:

Application Process

1. Online application: You can apply directly through LinkedIn/ by uploading your CV. In case you wish to submit your application via another format like audio/video, please, contact - UKI.recruitment@tcs.com
2. Skill-Based discussion: This will be a level 1 interview with the project team, it can be via video or in-person. Details will be confirmed by your recruiter.
3. Managerial discussion: This discussion will focus on behavioural aspects and person-organisation fit.
4. HR Discussion: This will be with one of the members of the HR team and will cover your career journey, aspirations for growth, compensation and any other questions you may have.

Beware of Fraudulent offers

This is to notify you that TCS does not ask for any sort of payment or security deposit from candidates at any stage of the recruitment process. The firm never sends out job offers from free internet email services like Gmail, Yahoo Mail, and so on. TCS has not authorised any third-party company to collect money on their behalf. As a vigilant job seeker, beware of fraudulent recruitment activity and protect your interests! You can write to UKI.recruitment@tcs.com to report any fraudulent activity.

Due to the high volume of applications, we will be unable to contact each applicant individually on the status of their application. If you have not received a direct response within 30 days, then it should be deemed unsuccessful on this occasion.

Join us and do more of what matters. Apply online now.