Databricks Senior Developer

Crisil Integral IQ delivers solutions and analytics to top financial institutions, driving strategic transformation, risk optimization, and operational excellence. Our offerings across research, risk, lending, analytics and operations have empowered clients to navigate complex markets, mitigate risks and unlock new opportunities. Our domain expertise, innovative solutions, and future-ready technologies such as AI and data science give clients the confidence to accelerate growth and achieve sustainable competitive advantage. Our globally diverse workforce operates in the Americas, Asia-Pacific, Europe, Australia and the Middle East.

We are looking for a Databricks Senior Developer who will join our Team supporting one of the biggest investment banks.

Key responsibilities:

Design and Develop Data Pipelines

• Build scalable, high-performance data pipelines using Databricks, PySpark, and SQL.
• Develop and optimize ETL/ELT workflows for batch and streaming data processing.

Lead Cloud Migration Initiatives

• Drive and support the migration of on-premises data platforms to cloud environments (preferably Azure).
• Refactor legacy data pipelines and ensure seamless transition to modern cloud-based architectures.
• Convert and optimize legacy SQL dialects into Spark SQL for improved performance and scalability.

Data Platform Engineering

• Develop and manage data solutions within the Databricks Lakehouse architecture.
• Ensure reliability, scalability, and performance of data processing workloads in production environments.

Collaboration & Stakeholder Engagement

• Work closely with data architects, analysts, and business stakeholders to understand data requirements and deliver solutions.
• Provide technical guidance and mentorship to junior engineers.

Data Quality & Governance

• Implement data quality frameworks, validation rules, and monitoring mechanisms.
• Ensure data accuracy, consistency, and compliance across pipelines.

CI/CD and DevOps Practices

• Design and maintain CI/CD pipelines for data engineering workflows.
• Apply best practices for version control, automated testing, and deployment.

Workflow Orchestration

• Design, implement, and maintain orchestration pipelines using tools such as: Databricks Workflows, Apache Airflow, dbt.
• Monitor job performance and troubleshoot failures proactively.

Cloud Integration & Optimization

• Integrate Databricks with Azure services (e.g., Data Factory, Azure Functions, storage accounts).
• Optimize cluster configurations, job execution, and cost utilization.

Performance Tuning & Optimization

• Analyze and optimize queries, Spark jobs, and data models for efficiency and reduced processing time.
• Implement best practices for partitioning, caching, and data storage formats.

Documentation & Best Practices

• Maintain clear technical documentation for pipelines, workflows, and system architecture.
• Promote coding standards, reusable components, and best engineering practices.

Skills required:

• 5 to 7 years of data engineering experience, with at least 3+ years of hands-on Databricks experience in production environments.
• Migration Expertise
• Proven experience in migrating on-premises data systems to cloud platforms.
• Strong proficiency in Python, PySpark, and SQL.
• Experience in converting legacy SQL dialects to Spark SQL is a plus.
• Familiarity with cloud computing concepts, preferably within Azure ecosystems.
• Experience integrating cloud functions or runtime services with data platforms is desirable.
• Solid understanding of CI/CD, data quality frameworks, and orchestration tools such as Databricks Workflows, Airflow, or dbt.

What we offer:

• Stable Employment in an International Company: Be part of a globally recognized organization that values stability and growth.
• Secure Employment Contract: Enjoy the peace of mind that comes with a solid employment contract.
• Flexible Hybrid Work Model: Experience the best of both worlds with our hybrid work model, allowing you to work from the office just 3 times a week and enjoy the flexibility of remote work the rest of the time.
• Comprehensive Benefits Package.
• Employee Referral Program: Help us grow our team and get rewarded through our Employee Referral Program.
• Education Reimbursement Policy: Invest in your future with our education reimbursement policy, supporting your continuous learning and professional development.
• Career Development Opportunities: Explore numerous internal and external opportunities to advance your career within the Finance Domain. We are committed to your growth and success.