Python Developer with Azure Databricks

Project description

You will join the team behind an internal AI platform for processing and interacting with unstructured data. The team is currently over 30 people strong and is organized into agile teams, each of which is self-sufficient and handles the creation of features from the idea stage, through analysis, implementation, testing, production deployment, and maintenance. The team is international, and it's located in Krakow, Wroclaw and London

Responsibilities

• Maintain and enhance the existing Python-based reporting system, ensuring stability and accuracy in regulatory reporting.
• Monitor, troubleshoot, and resolve issues in the current reporting workflows.
• Ensure that reports comply with regulatory requirements and meet audit standards.
• Lead the migration of the existing reporting system to Azure Databricks, ensuring a smooth transition.
• Design and implement new, scalable reporting solutions using PySpark, SQL, and Databricks visualization tools.
• Optimize data processing pipelines to enhance performance, reliability, and efficiency of regulatory reports.
• Optimize PySpark performance to handle large-scale financial data processing for regulatory reporting.
• Improve SQL query execution to ensure fast, efficient report generation.
• Implement best practices for structured and unstructured data processing.
• Automate reporting processes to reduce manual intervention and enhance accuracy.
• Work with DevOps teams to ensure seamless Azure Databricks integration into the existing infrastructure.

Skills

Must have

• Databricks Concepts
• Databricks performance optimizations
• ACID properties of RDMS
• Python, PySpark
• SQL Concepts
• Databricks Queries and Visualization
• Exposure to CI/CD DevOps Pipelines

Nice to have

• Azure Data Services
• ETL & Data Processing,
• Financial Data Processing, experience handling risk, compliance, and regulatory data in an investment banking environment.
• Data Modeling, meta data management
• Git, GitHub, GitLab,
• Jenkins
• Regulatory compliance knowledge: Basel, MiFID, GDPR
• Big Data
• Cloud security and access controls (IAM, RBAC)
• Familiarity with Docker, Kubernetes, Apache