Data Engineer
Sheffield, South Yorkshire, United Kingdom
Teksystems
Job Title: Data Engineer (w. Prophecy) In this role you will: Design, develop, and optimize data pipelines using Azure Databricks, PySpark, and Prophesy. Implement and maintain ETL/ELT pipelines using Azure Data Factory (ADF) and Apache Airflow for orchestration. Develop and optimize complex SQL queries … and Python-based data transformation logic. Work with version control systems(GitHub, Azure DevOps) to manage code and deployment processes. Automate deployment of data pipelines using CI/CD practices in Azure DevOps. Ensure data quality, security, and compliance with best practices. Monitor and … role you should meet the following requirements: Must have experience with Delta Lake and Lakehouse architecture. Proven experience in data engineering, working with Azure Databricks, PySpark, and SQL. Hands-on experience with Prophesy for data pipeline development. Proficiency in Python for data processing and transformation. Experience with Apache More ❯
Employment Type: Contract
Rate: £250 - £600/day
Posted: