3 of 3 Remote Apache Airflow Jobs in Lancashire

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. ...

management. Requirements for the Principal Data Engineer: Experience as a lead or principal data engineer. Prior experience designing data platform(s) within GCP, across Airflow, Big Query, Data Flow, Data Fusion, and Data Stream. Deep understanding of Data Mesh design and Data Lake/Warehouse solutions. Previously led teams ...

experienced team members and fostering a culture of knowledge sharing Communicating clearly and effectively with diverse technical and non-technical audiences Tech stack Python , Airflow , Terraform Azure (but open to other cloud experience) DBT (nice to have) Exposure to ML pipelines beneficial ...