Data Engineer

NO VISA REQUIREMENTS

Location: Glasgow (3+ days)Reports to: Head of IT My client is undergoing a major transformation of their entire data landscape-migrating from legacy systems and manual reporting into a modern Azure + Databricks Lakehouse. They are building a secure, automated, enterprise-grade platform powered by Lakeflow Declarative Pipelines, Unity Catalog and Azure Data Factory. They are looking for a Mid-Level Data Engineer to help deliver high-quality pipelines and curated datasets used across Finance, Operations, Sales, Customer Care and Logistics.

• Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL.
• Work within a Medallion architecture (Bronze Silver Gold) to deliver reliable, high-quality datasets.
• Ingest data from multiple sources including ChargeBee, legacy operational files, SharePoint, SFTP, SQL, REST and GraphQL APIs using Azure Data Factory and metadata-driven patterns.
• Apply data quality and validation rules using Lakeflow Declarative Pipelines expectations.

• Develop clean and conforming Silver & Gold layers aligned to enterprise subject areas.
• Contribute to dimensional modelling (star schemas), harmonisation logic, SCDs and business marts powering Power BI datasets.
• Apply governance, lineage and permissioning through Unity Catalog.

• Use Lakeflow Workflows and ADF to orchestrate and optimise ingestion, transformation and scheduled jobs.
• Help implement monitoring, alerting, SLAs/SLIs and runbooks to support production reliability.
• Assist in performance tuning and cost optimisation.

• Contribute to CI/CD pipelines in Azure DevOps to automate deployment of notebooks, Lakeflow Declarative Pipelines, SQL models and ADF assets.
• Support secure deployment patterns using private endpoints, managed identities and Key Vault.
• Participate in code reviews and help improve engineering practices.

• Work with BI and Analytics teams to deliver curated datasets that power dashboards across the business.
• Contribute to architectural discussions and the ongoing data platform roadmap.

• Databricks: Lakeflow Declarative Pipelines, Lakeflow Workflows, Unity Catalog, Delta Lake
• Azure: ADLS Gen2, Data Factory, Event Hubs (optional), Key Vault, private endpoints
• Languages: PySpark, Spark SQL, Python, Git
• DevOps: Azure DevOps Repos & Pipelines, CI/CD
• Analytics: Power BI, Fabric

• Commercial and proven data engineering experience.
• Hands-on experience delivering solutions on Azure + Databricks.
• Strong PySpark and Spark SQL skills within distributed compute environments.
• Experience working in a Lakehouse/Medallion architecture with Delta Lake.
• Understanding of dimensional modelling (Kimball), including SCD Type 1/2.
• Exposure to operational concepts such as monitoring, retries, idempotency and backfills.

• Keen to grow within a modern Azure Data Platform environment.
• Comfortable with Git, CI/CD and modern engineering workflows.
• Able to communicate technical concepts clearly to non-technical stakeholders.
• Quality-driven, collaborative and proactive.

• Databricks Certified Data Engineer Associate.
• Experience with streaming ingestion (Auto Loader, event streams, watermarking).
• Subscription/entitlement modelling (e.g., ChargeBee).
• Unity Catalog advanced security (RLS, PII governance).
• Terraform or Bicep for IaC.
• Fabric Semantic Models or Direct Lake optimisation experience.

• Opportunity to shape and build a modern enterprise Lakehouse platform.
• Hands-on work with Azure, Databricks and leading-edge engineering practices.
• Real progression opportunities within a growing data function.
• Direct impact across multiple business domains.