Data Engineer

Employment Type: engagement is inside IR‑35 through an umbrella company

Requirements 'must have':

• Education: Bachelor’s degree in Computer Science, Engineering, or a related field (or equivalent experience).
• 4+ years of experience developing data pipelines and data warehousing solutions using Python and libraries such as Pandas, NumPy, PySpark, etc.
• 3+ years hands-on experience with cloud services, especially Databricks, for building and managing scalable data pipelines
• 3+ years of proficiency in working with Snowflake or similar cloud-based data warehousing solutions
• 3+ years of experience in data development and solutions in highly complex data environments with large data volumes.
• Solid understanding of ETL principles, data modelling, data warehousing concepts, and data integration best practices-Familiarity with agile methodologies and the ability to work collaboratively in a fast-paced, dynamic environment.
• Experience with code versioning tools (e.g., Git)
• Knowledge of Linux operating systems
• Familiarity with REST APIs and integration techniques
• Familiarity with data visualization tools and libraries (e.g., Power BI)
• Background in database administration or performance tuning
• Familiarity with data orchestration tools, such as Apache Airflow
• Previous exposure to big data technologies (e.g., Hadoop, Spark) for large data processing
• Strong analytical skills, including a thorough understanding of how to interpret customer business requirements and translate them into technical designs and solutions.
• Strong communication skills both verbal and written. Capable of collaborating effectively across a variety of IT and Business groups, across regions, roles and able to interact effectively with all levels.
• Self-starter. Proven ability to manage multiple, concurrent projects with minimal supervision. Can manage a complex ever changing priority list and resolve conflicts to competing priorities.
• Strong problem-solving skills. Ability to identify where focus is needed and bring clarity to business objectives, requirements, and priorities.

Requirements 'nice to have':

• Experience in financial services
• Knowledge of regulatory requirements in the financial industry

Tasks:

• Collaborating with cross-functional teams to understand data requirements, and design efficient, scalable, and reliable ETL processes using Python and Databricks
• Developing and deploying ETL jobs that extract data from various sources, transforming them to meet business needs.
• Taking ownership of the end-to-end engineering lifecycle, including data extraction, cleansing, transformation, and loading, ensuring accuracy and consistency.
• Creating and managing data pipelines, ensuring proper error handling, monitoring and performance optimizations
• Working in an agile environment, participating in sprint planning, daily stand-ups, and retrospectives.
• Conducting code reviews, providing constructive feedback, and enforcing coding standards to maintain a high quality.
• Developing and maintaining tooling and automation scripts to streamline repetitive tasks.
• Implementing unit, integration, and other testing methodologies to ensure the reliability of the ETL processes
• Utilizing REST APIs and other integration techniques to connect various data sources
• Maintaining documentation, including data flow diagrams, technical specifications, and processes.
• Designing and implementing tailored data solutions to meet customer needs and use cases, spanning from streaming to data lakes, analytics, and beyond within a dynamically evolving technical stack.
• Collaborate seamlessly across diverse technical stacks, including Databricks, Snowflake, etc.
• Developing various components in Python as part of a unified data pipeline framework.
• Contributing towards the establishment of best practices for the optimal and efficient usage of data across various on-prem and cloud platforms.
• Assisting with the testing and deployment of our data pipeline framework utilizing standard testing frameworks and CI/CD tooling.
• Monitoring the performance of queries and data loads and perform tuning as necessary.
• Providing assistance and guidance during QA & UAT phases to quickly confirm the validity of potential issues and to determine the root cause and best resolution of verified issues.
• Adhere to Agile practices throughout the solution development process.
• Design, build, and deploy databases and data stores to support organizational requirements.