Senior Data Engineer – Python, Databricks & Counterparty Credit Risk
Senior Data Engineer – Python, Databricks & Counterparty Credit Risk
Location: London – Hybrid
We are recruiting for an experienced Senior Data Engineer to join a leading global banking organisation, working across a highly technical data environment supporting Counterparty Credit Risk (CCR), Credit Risk and wider risk data.
This is a hands-on engineering position requiring strong expertise across Python, PySpark, SQL and Databricks, alongside experience designing, building and optimising scalable data pipelines and Lakehouse solutions.
Candidates should ideally have previously worked with Counterparty Credit Risk or Credit Risk data, including areas such as counterparty/exposure data, risk analytics, regulatory risk or associated banking risk datasets.
Key Responsibilities
- Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks
- Build and optimise ETL/ELT workflows using Spark and Delta Lake
- Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions
- Work with datasets relating to areas such as counterparties, credit exposures, limits, collateral and risk analytics
- Design robust data models and architectures across Lakehouse / Medallion environments
- Develop and manage Databricks notebooks, jobs and workflows
- Optimise large-scale distributed data processing and troubleshoot Spark performance
- Implement strong data quality, reconciliation, governance and monitoring controls
- Build reliable logging and alerting across production data pipelines
- Integrate data platforms with Power BI dashboards and wider applications
- Support CI/CD and engineering best practices across development, testing and production environments
- Work closely with Risk, Technology, Analytics and other senior stakeholders
Required Experience
- 10+ years' experience across Data Engineering or closely related data disciplines
- Strong hands-on Python development experience
- Strong PySpark / Apache Spark engineering experience
- Extensive hands-on experience with Databricks, including notebooks, jobs/workflows and Delta Lake
- Advanced SQL and relational database experience
- Strong track record designing and building scalable ETL/ELT data pipelines
- Experience with Delta Lake, Lakehouse and/or Medallion architectures
- Experience handling banking risk data, ideally within Counterparty Credit Risk (CCR), Credit Risk, counterparty exposure or closely related risk domains
- Strong understanding of data modelling, data quality and production data engineering
- Cloud experience across Azure, AWS and/or GCP
- Experience with modern data platforms/warehouses such as Snowflake, BigQuery or Redshift
- Experience integrating data platforms with Power BI
- Experience with orchestration such as Databricks Workflows or Airflow
Highly Desirable
- Direct experience engineering data for Counterparty Credit Risk, including exposure/counterparty datasets
- Knowledge of areas such as PFE, EPE, EAD, CVA/XVA, SA-CCR, collateral, netting or counterparty limits
- Previous experience within an investment bank, global bank or capital-markets environment
- Databricks Data Engineer certification
- Kafka / Structured Streaming experience
- Delta Lake CDC, Time Travel and optimisation
- CI/CD, Git and DevOps experience
- Docker / Kubernetes experience
- Experience exposing data through APIs or supporting React-based front-end applications
- Experience supporting analytics or ML workloads within Databricks
We are particularly interested in genuinely hands-on Senior Data Engineers who can demonstrate the Databricks and Python solutions they have personally designed, built, optimised and supported in production.
If you have strong Databricks, Python and PySpark engineering experience alongside exposure to Counterparty Credit Risk, Credit Risk or banking risk data, please apply for further information.