Data Engineer

We are seeking a highly skilled Python Data Engineer with hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimization, and Azure cloud services. This role involves designing, developing, and deploying scalable data processing solutions in a containerized environment, with an emphasis on maintainable, configurable, and test-driven code delivery.

Key Responsibilities:

• Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark.
• Implement unit and behavior-driven testing using Behave, ensuring robust mocking and patching of dependencies.
• Design and maintain Delta Lake tables for optimized query performance, ACID compliance, and incremental data loads.
• Build and manage containerized environments using Docker for consistent development, testing, and deployment.
• Develop configurable, parameter-driven codebases to support modular and reusable data solutions.
• Integrate Azure services, including Azure Functions for serverless transformation logic, Azure Key Vault for secure credential management, and Azure Blob Storage for data lake operations.
• Collaborate closely with cloud architects, data scientists, and DevOps teams to ensure seamless CI/CD workflows, version control, and environment consistency.
• Troubleshoot and optimize Spark jobs for performance and scalability in production environments.
• Maintain technical documentation and adhere to best practices in cloud security and data governance.

Required Skills and Experience:

• Strong proficiency in Python programming with emphasis on modular and test-driven design.
• Demonstrated experience in writing unit tests and BDD scenarios using Behave or similar frameworks.
• In-depth understanding of mocking, patching, and dependency injection in Python testing.
• Proficiency in PySpark with hands-on experience in distributed data processing and performance tuning.
• Solid understanding of Delta Lake concepts, transactional guarantees, and schema evolution.
• Experience with Docker for development, testing, and deployment workflows.
• Familiarity with Azure components such as Azure Functions, Key Vault, Blob Storage, and Data Lake Storage Gen2.
• Ability to implement configuration-driven applications for flexible deployment across environments.
• Experience with CI/CD pipelines (Azure DevOps or similar) and infrastructure-as-code tools is a plus.
• Strong problem-solving skills and ability to work independently in fast-paced, agile environments.

Preferred Qualifications:

• Experience developing in Databricks or Synapse with Delta Lake integration.
• Knowledge of best practices in data security and governance within Azure ecosystems.
• Strong communication skills and experience collaborating with distributed teams.