Databricks Architect

Job Title: Databricks Architect

Location: UK/Europe - Remote

Engagement Type: Contract (Outside IR35)

Role Overview

We are seeking a Senior Databricks Architect with 15+ years of experience in Data Engineering/Analytics to design and deliver scalable, secure, and cost-efficient data solutions on AWS. This is a leadership role requiring strong hands-on expertise, proactive risk management, and excellent stakeholder communication skills.

Key Responsibilities

• Architect and implement data solutions using Databricks on AWS, PySpark, and Python
• Lead cloud migration initiatives, including on-prem to AWS migrations
• Drive GenAI-enabled automation for data asset migration
• Develop scalable data models and deliver actionable business insights
• Design and optimize ETL pipelines (Informatica or similar tools)
• Build and deploy PySpark applications for large-scale data processing
• Work on LLM pretraining, fine-tuning, augmentation, and optimization
• Ensure high data quality through cleaning, transformation, and validation (Python/SQL)
• Automate data workflows to enhance scalability and efficiency
• Maintain clear technical documentation and best practices

Required Experience

• 15+ years in Data Analyst/Data Engineer/Data Architect roles
• Strong expertise in Databricks on AWS
• Advanced Python (Pandas, Scikit-learn) and SQL skills
• Proven experience in large-scale cloud data migrations
• Hands-on experience with LLMs and GenAI-based automation
• Strong experience designing secure, high-performance data architectures

Key Competencies

• Proactive risk & issue management
• Strong requirement clarification and stakeholder engagement skills
• Excellent communication – able to translate technical insights into business value
• Critical thinker with strong problem-solving capabilities
• Self-driven, autonomous, and collaborative approach