Job Title: Platform Engineer Databricks Modernisation
Leeds, West Yorkshire, Yorkshire, United Kingdom
Hybrid / WFH Options
Hybrid / WFH Options
Unolabs
are looking for an experienced Platform Engineer to lead the infrastructure and CI/CD migration, build and promote Dockerized microservices, and enable a scalable deployment strategy using Terraform, GitLab, and Jenkins. Youll collaborate with Data engineers, Data stakeholders, and DevOps specialists to ensure smooth migration of pipelines. Key Responsibilities: Infrastructure Migration & Automation Split and migrate codebase into sub-repos … bucket permissions, and Jenkins agents. Ensure the new infrastructure supports automated deployment pipelines for 8,000+ test cases, notebooks, and services. CI/CD Pipeline Development Develop and enhance GitLab CI/CD pipelines and integrate them with Jenkins where applicable. Enable Docker-in-Docker strategies for GitLab runners to support test and integration pipelines, while considering container security risks … images for: Databricks compute environments. Supporting microservices such as Broker, Scheduler, and Status Monitor. Push and manage Docker images in AWS ECR (Elastic Container Registry) and integrate them with GitLab CI/CD runners and ECS-based compute in CDP. Notebook & Pipeline Migration Migrate critical pipelines (analyst-developed, exploratory and transformation notebooks) from DPS to CDP using Code Promotion Process More ❯
Employment Type: Contract, Work From Home
Posted: