Senior Data Engineer - Azure Databricks/ SC Clearance
London, South East, England, United Kingdom
Hybrid/Remote Options
Hybrid/Remote Options
Crimson
from APIs, databases, and financial data sources into Azure Databricks. Optimize pipelines for performance, reliability, and cost, incorporating data quality checks. Develop complex transformations and processing logic using Spark (PySpark/Scala) for cleaning, enrichment, and aggregation, ensuring accuracy and consistency across the data lifecycle. Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related services. Apply best practices for development, deployment, and workload optimization. Program in SQL, Python, R, YAML, and JavaScript. Integrate data from relational databases, APIs, and streaming sources using best-practice patterns. Collaborate with API developers for seamless data exchange. Utilize Azure Purview for governance and quality monitoring. Implement lineage tracking, metadata management, and compliance More ❯
Employment Type: Contractor
Rate: £600 - £620 per day
Posted: