Data Engineer

As a Data Engineer, you will be responsible for:

Data Engineering & Development

• Design, build, and maintain high-quality, scalable, and tested data pipelines.
• Develop and manage Databricks structured streaming pipelines.
• Build and optimize event-driven and real-time data processing solutions.
• Implement and maintain Unity Catalog-based Lakehouse architecture.
• Develop analytics-ready datasets to support business insights and reporting.

Platform & Automation

• Build and manage CI/CD pipelines using Azure DevOps.
• Identify and implement automation opportunities across workflows.
• Ensure reliable and stable data platform operations.
• Apply governance, security, and documentation standards.

Data Quality & Reliability

• Establish the Data Lakehouse as a trusted and reliable source of truth.
• Monitor, troubleshoot, and resolve data incidents.
• Support business users and technical teams with data-related queries.
• Continuously improve platform performance and reliability.

Collaboration & Support

• Work closely with data science, analytics, platform, and business teams.
• Champion data engineering best practices.
• Provide technical guidance and mentorship where required.
• Contribute to a culture of learning, quality, and continuous improvement.

Essential Skills

• Strong experience with Azure Databricks and cloud data platforms.
• Advanced proficiency in Python, PySpark, and SQL.
• Experience developing Spark/Databricks pipelines.
• Hands-on experience with structured streaming and event-driven systems.
• Strong understanding of Lakehouse architecture and best practices.
• Experience with Unity Catalog.
• Expertise in Azure DevOps and CI/CD pipelines.
• Knowledge of data modelling (dimensional/star schemas).
• Experience working in Agile environments.

Desirable Skills

• Exposure to multiple data technology stacks.
• Experience in large-scale enterprise environments.
• Knowledge of security, governance, and compliance frameworks.