Azure Data Engineer
Manchester, North West, United Kingdom
Hybrid / WFH Options
Hybrid / WFH Options
Agile Recruit
including streaming and batch data Develop and maintain data transformation workflows using PySpark, Scala, and SQL on Databricks Data Platform & Optimization Design and optimize Databricks clusters for performance, cost management, and resource utilization Implement medallion architecture (Bronze, Silver, Gold) for data processing and transformation Optimize Spark jobs and queries for maximum performance and cost efficiency Monitor and troubleshoot data … Mastery: Deep expertise in Azure data services including: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Azure SQL Database Databricks Proficiency: Advanced skills in Databricks including cluster management and optimization Programming: Strong proficiency in Python, Scala, and SQL for data processing, transformation, and performance optimization Big Data Technologies: Proven experience with distributed computing and big data frameworks … Streaming Experience with event-driven architectures and streaming data pipelines Data Governance Understanding of data governance frameworks, Unity Catalog, and data quality practices Experience with data lineage, cataloguing, and metadata management Desirable Skills Experience with other cloud platforms (AWS, GCP) Knowledge of containerization technologies (Docker, Kubernetes) Familiarity with machine learning workflows and MLOps practices Experience with data visualization More ❯
Employment Type: Contract
Rate: £450 - £550/day
Posted: