Databricks Consultant
Data Engineer (Azure & Databricks) - 6-month Contract
We are seeking an experienced Senior Data Engineer with strong Databricks expertise to design, build, and maintain a scalable cloud-based data platform for a private equity firm. This is a 6-month contract based in London with hybrid working.
Key Responsibilities
- Design and build robust data pipelines using Azure Data Factory and Databricks
- Develop ETL/ELT processes to transform raw data into analytics-ready datasets
- Architect and maintain data lake solutions using Azure Data Lake Storage
- Implement Databricks solutions using Delta Lake and medallion architecture (bronze, silver, gold)
- Optimise performance, data quality, and cost efficiency across pipelines and clusters
- Set up monitoring, alerting, and ensure high availability of data services
- Collaborate with data teams and stakeholders to deliver business-focused solutions
Essential Skills & Experience
- Strong hands-on experience with Databricks (essential)
- Proven experience with Azure services
- Proficiency in Python and SQL
- Experience with CI/CD, Git, and deployment pipelines
- Solid understanding of data modelling and data architecture
- Knowledge of data governance and security best practices
- Experience with Terraform or Infrastructure-as-Code
- Azure DevOps
- Relevant Azure or Databricks certifications