4 of 4 Jupyter Jobs in the North East

solutions Data processing & orchestration Apache Spark - understanding of distributed data processing architecture and execution Apache Airflow - configuring DAGs and managing task orchestration at scale Jupyter Notebooks - for exploratory data analysis and pipeline prototyping Understanding of dimensional data models and slowly changing dimensions (SCD Types 1, 2, 3) Data analysis skills ...

Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs Use of Jupyter notebooks and/or Amazon Athena to query and validate data Data analysis to identify root cause of issues and familiarity with basic data structures ...

solutions Data processing & orchestration Apache Spark - understanding of distributed data processing architecture and execution Apache Airflow - configuring DAGs and managing task orchestration at scale Jupyter Notebooks - for exploratory data analysis and pipeline prototyping Understanding of dimensional data models and slowly changing dimensions (SCD Types 1, 2, 3) Data analysis skills ...

deployment processes Integrate security scanning agents into the development and deployment pipeline to ensure code and infrastructure meet security standards Develop and maintain notebooks (Jupyter) to support data exploration, analysis, and prototyping Build and maintain integrations with DWP Integration Team products, ensuring reliable and secure data exchange Collaborate with cross … with Docker and containerisation Experience building and maintaining CI/CD pipelines using GitLab Experience integrating security scanning tools into development pipelines Experience with Jupyter Notebooks for data exploration and analysis Experience integrating with third-party or internal platform products, ideally within a government or DWP context Understanding of secure ...