Senior Data / AI Engineer

Senior Data / AI Engineer

Location: London (Hybrid, 2 days per week onsite)
Salary: Up to £85,000

The Opportunity

We are seeking a Senior Data / AI Engineer to play a key role in building and scaling innovative data products. This is more than a traditional data engineering position. You will work closely with product, commercial and technology teams to transform raw and third-party data into valuable, customer-facing solutions.

The role combines modern data engineering, AI capabilities and product thinking, offering the opportunity to shape how data is monetised, enriched and delivered to end users.

Key Responsibilities Data Product Development

• Design and build scalable data products that support commercial and customer-facing solutions.
• Transform raw and third-party data into structured, enriched and product-ready datasets.
• Collaborate with product and commercial stakeholders to define how data is packaged, accessed and delivered.
• Develop data delivery solutions through APIs, internal tools and customer-facing platforms.

AI & Data Enrichment

• Apply AI and LLM technologies to enrich datasets through classification, tagging, summarisation and insight generation.
• Design and develop LLM-powered product features.
• Implement Retrieval-Augmented Generation (RAG) solutions to enhance data quality and unlock new product capabilities.
• Support the development of AI-driven solutions including intelligent search, recommendation engines and insights platforms.
• Ensure AI systems are scalable, resilient and governed appropriately.

Platform Engineering

• Support the deployment, operation and lifecycle management of data and AI platforms.
• Own and evolve the Azure Databricks-based data platform architecture to support business growth.
• Drive performance and cost optimisation across Azure services through monitoring, tuning and automation.
• Build and maintain reliable batch and streaming data pipelines.
• Implement robust testing, observability and data quality controls.
• Promote engineering best practices through CI/CD, automation and platform standardisation.

Stakeholder Engagement

• Partner with business and product stakeholders to identify high-value data opportunities.
• Translate business requirements into scalable technical solutions.
• Act as a bridge between commercial, product and engineering teams.
• Support prioritisation decisions based on business value and impact.

Leadership

• Contribute to the wider data and AI strategy.
• Mentor junior engineers and promote engineering excellence.
• Help build a high-performing team with a strong product and commercial mindset.

Required Skills & Experience Data Engineering & Cloud

• Extensive experience designing, building and operating production data platforms using Azure Databricks and Azure.
• Strong hands-on expertise with Apache Spark, PySpark, Spark SQL and Delta Lake.
• Experience implementing and managing Unity Catalog, including governance, lineage and access controls.
• Proven delivery of lakehouse and Medallion architectures.
• Advanced Python and SQL skills for scalable, production-grade solutions.
• Experience building modern batch and real-time data pipelines.
• Strong understanding of API design and data product delivery mechanisms.
• Experience deploying Databricks solutions using CI/CD pipelines, Azure DevOps and/or GitHub Actions.
• Experience integrating Databricks with ADLS Gen2, Azure Identity, Key Vault and external data sources.

Data Product Experience

• Experience building customer-facing data products or commercial data solutions.
• Strong understanding of how data can be structured, exposed and scaled for end users.
• Ability to focus on measurable business outcomes rather than solely technical delivery.
• Experience working closely with product, commercial or customer-facing teams.

AI & Machine Learning

• Practical experience with Generative AI and LLM technologies such as Azure OpenAI and LangChain.
• Experience applying AI solutions to real-world business challenges.
• Knowledge of embeddings, vector databases and semantic search.
• Experience working with unstructured data including text and document-based datasets.
• Understanding of AI governance, monitoring and ethical considerations.

Desirable Experience

• Airbyte or similar modern data ingestion platforms.
• Terraform and Infrastructure as Code.
• LLM frameworks such as LangChain or LlamaIndex.
• Experimentation, testing or A/B testing frameworks.
• Experience building AI-enabled or data-driven products.
• Background within Life Sciences, Healthcare or highly regulated industries.

About You

• Product-focused and driven by business impact.
• Equally comfortable discussing technical solutions and commercial objectives.
• Passionate about delivering production-ready solutions rather than proofs of concept.
• Pragmatic, collaborative and delivery-oriented.
• Naturally curious about emerging AI technologies and how they can create commercial value.