Data Engineering Lead
Data Engineering Lead Certain Advantage are hiring for a Data Engineering Lead based in London.This role is on an initial contract till the end of the year and is required to be onsite 3 days a week.The successful candidate will lead the data acquisition function by defining ingestion strategy and building Azure-based data pipelines.You will translate end-user requirements into scalable engineering solutions, act as the domain’s primary Data Manager for data governance, lineage, quality, and master data management, and lead a team of data engineers in delivering reliable, compliant, and high-value data solutions to Trading, Risk, Finance, and Operations.Your responsibilities:
- Own ingestion strategy and define scalable batch/streaming data patterns
- Design and deliver Azure-based data pipelines and platform solutions
- Standardize connectors, schemas, controls, CI/CD, IaC, and automated testing
- Lead onboarding of enterprise and market data sources with SLAs and observability
- Partner with Product Owners, Architects, and business stakeholders on data design and governance
- Act as domain Data Manager for lineage, metadata, quality, access, and master data ownership
- Ensure compliance with data lifecycle, audit, records management, and governance standards
- Drive engineering best practices, operational excellence, and continuous improvement
- Manage capacity planning, skills development, and vendor contributions
- Expertise in Azure data services (ADLS, ADF, Databricks, Synapse, Event Hub/Kafka), Python, SQL, Spark/PySpark.
- Experience with data modeling, metadata and lineage tooling; performance & cost optimization.
- Proven leadership in engineering teams; Agile/DevOps ways of working; CI/CD; test automation.
- Exposure to energy trading systems (ETRM/Endur), market data (ICE, Platts, Bloomberg), SAP.
- Degree in Computer Science/Engineering or equivalent experience; Azure Data Engineer or Databricks certifications a plus.