document best practices for managing and maintaining robust, scalable data pipelines. Data Governance & Security: Implement and maintain data governance policies using Unity Catalog, ensuring proper organization, access control, and metadatamanagement across data assets. Ensure data security best practices, such as encryption at rest and in transit, and role-based access control (RBAC) within Azure Databricks and Azure … to minimize processing time and resource consumption. Continuously monitor and improve pipeline performance, addressing bottlenecks and optimizing for cost efficiency in Azure. Automation & Monitoring: Automate data pipeline deployment and management using tools like Terraform, ensuring consistency across environments. Set up monitoring and alerting mechanisms for pipelines using Databricks built-in features and Azure Monitor to detect and resolve issues More ❯
Technical knowledge and underrating of the following is required:- Data Engineering Best Practices, CI/CD for data pipelines, Modular, testable ETL development, Git-based collaboration and branching strategies, Metadatamanagement and data documentation, Reusability and multi-tenancy in data models and Orchestrating data through medallion architecture or equivalent You will ideally have strong communication & coaching skills as More ❯
Cwmbran, Gwent, Wales, United Kingdom Hybrid / WFH Options
Hays
DETAILS - £400-£450 per day - Outside IR35 - Remote role based in South Wales - 3-month contract with extension - Immediate start SKILLS - Strong experience with SQL Database. - Excellent knowledge of Metadata management. - Extensive experience in Data Governance and AI. - Data Literacy awareness and IT Training. RESPONSIBILITIES - Develop a plan/strategy for improving the organisation's collection, storage and use More ❯
Croesyceiliog, Gwent, United Kingdom Hybrid / WFH Options
Hays Technology
DETAILS - 400- 450 per day - Outside IR35 - Remote role based in South Wales - 3-month contract with extension - Immediate start SKILLS - Strong experience with SQL Database. - Excellent knowledge of Metadata management. - Extensive experience in Data Governance and AI. - Data Literacy awareness and IT Training. RESPONSIBILITIES - Develop a plan/strategy for improving the organisation's collection, storage and use More ❯