Data Engineer (SC Cleared, Databricks)

Senior Data Engineer (Azure Databricks) - SC Cleared

Location: London, UK (3 days per week onsite)

Salary: Up to £75,000 per annum

Job Type: Permanent, Full-Time

Security Clearance: Active SC Clearance Required

The Opportunity

We are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities.

This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices. You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights.

Key Responsibilities

Data Pipeline Development & Optimisation

• Design, develop, and maintain scalable and reliable data pipelines.
• Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers.
• Optimise pipeline performance, scalability, and cost efficiency.
• Implement data quality controls, validation processes, and monitoring solutions.

Data Transformation & Processing

• Develop complex data transformation logic using Spark (PySpark or Scala).
• Build data cleansing, enrichment, aggregation, and processing solutions.
• Ensure data accuracy, consistency, and reliability throughout the data life cycle.

Azure Databricks Development

• Develop and support solutions within Azure Databricks.
• Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services.
• Apply Databricks best practices for development, deployment, and workload optimisation.
• Develop solutions using SQL, Python, R, YAML, and JavaScript.

Data Integration

• Integrate data from relational databases, APIs, and streaming platforms.
• Design and implement robust data integration patterns.
• Collaborate with development teams to support seamless data exchange across systems.

Data Quality & Governance

• Utilise Azure Purview for data governance and data quality initiatives.
• Implement data lineage, metadata management, monitoring, and alerting capabilities.
• Ensure compliance with organisational data governance standards and best practices.

Collaboration & Delivery

• Work closely with data scientists, analysts, economists, and technical stakeholders.
• Translate business requirements into scalable technical solutions.
• Participate in code reviews, technical discussions, and knowledge-sharing sessions.
• Contribute to Agile delivery processes and continuous improvement initiatives.

DevOps & Automation

• Automate data engineering workflows and deployments.
• Develop and maintain CI/CD pipelines.
• Promote DevOps best practices across the wider engineering team.

Required Skills & Experience

• 8+ years of Data Engineering experience (10+ years preferred).
• Minimum 3 years of hands-on Azure Databricks experience.
• Strong expertise in Python, PySpark, and/or Scala.
• Extensive experience with:
• Azure Databricks
• Azure Data Factory
• Azure Blob Storage
• Azure SQL Database
• Delta Lake
• Spark SQL
• Strong understanding of:
• Data architecture
• Data modelling
• Data warehousing
• Data integration patterns
• Experience working with large-scale datasets and complex data pipelines.
• Knowledge of SQL and NoSQL databases.
• Experience with data governance, data quality frameworks, and metadata management.
• Experience using Git and working within Agile delivery environments.
• Excellent communication, stakeholder management, and problem-solving skills.

Desirable Skills

• Azure Purview
• Kafka
• Azure Event Hubs
• Power BI
• Tableau
• Azure DevOps
• Jenkins
• Docker
• Kubernetes
• Experience within Financial Services, Banking, or Economic Data environments
• Microsoft Certified: Azure Data Engineer Associate
• Databricks Certifications

What's On Offer

• Salary up to £75,000.
• Hybrid working model with 3 days per week in a London office.
• Opportunity to work on a large-scale Azure and Databricks data platform.
• Exposure to enterprise-level data engineering challenges and modern cloud technologies.
• Collaborative and technically focused environment.
• Career development and certification support opportunities.

Please note: Active SC Clearance is required for this position. Applicants without current SC Clearance will not be considered.