7 of 7 Apache Airflow Jobs in Lancashire

automation to ensure successful project delivery, adhering to client timelines and quality standards. Implement and manage real-time and batch data processing frameworks (e.g., Apache Kafka, Apache Spark, Google Cloud Dataproc) in line with project needs. Build and maintain robust monitoring, logging, and alerting systems for client projects … languages like Python, Bash, or Go to automate tasks and build necessary tools. Expertise in designing and optimising data pipelines using frameworks like Apache Airflow or equivalent. Demonstrated experience with real-time and batch data processing frameworks, including Apache Kafka, Apache Spark, or Google Cloud Dataflow. ...

strong plus Technical Skills: • Proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL) • Familiarity with data pipeline and workflow management tools (e.g., Apache Airflow) • Experience with programming languages such as Python, Java, or Scala. Python is highly preferred • Basic understanding of cloud platforms and services (e.g. ...

tuning. Hands-on expertise with Fivetran and dbt. Good understanding of data modelling, governance, and security best practices. Familiarity with orchestration tools such as Airflow or Prefect (advantageous). Experience working in Azure (or AWS/GCP). Strong analytical and collaboration skills, with great attention to detail. ...

scale. Deep understanding of distributed systems design and implementation. Strong Kubernetes experience managing production workloads. Proficiency with workflow orchestration frameworks such as Temporal, Airflow, or similar. Hands-on experience with AWS (Azure familiarity is a plus). Exposure to large-scale data frameworks such as Spark, Athena, or equivalents. ...

management. Requirements for the Principal Data Engineer: Experience as a lead or principal data engineer. Prior experience designing data platform(s) within GCP, across Airflow, Big Query, Data Flow, Data Fusion, and Data Stream. Deep understanding of Data Mesh design and Data Lake/Warehouse solutions. Previously led teams ...

experienced team members and fostering a culture of knowledge sharing Communicating clearly and effectively with diverse technical and non-technical audiences Tech stack Python , Airflow , Terraform Azure (but open to other cloud experience) DBT (nice to have) Exposure to ML pipelines beneficial ...

images to registries, and ensures secure and efficient AKS cluster deployments. Implements infrastructure as code using Terraform, including network, security, and compute resources. Supports Airflow as part of platform orchestration. Skills & Qualifications : Azure Landing Zone principles, vNet design, RBAC, networking, and hybrid connectivity. CI/CD tooling, especially GitLab ...