1 to 25 of 947 Apache Airflow Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

owned infrastructure. You'll work as part of a technical team building and supporting data pipelines using Python, Azure and Kubernetes, alongside technologies including Apache Airflow and Apache Spark. Key Responsibilities: Build and maintain new and existing data pipelines Develop data solutions using Python, Azure and Kubernetes … Work with Apache Airflow and Apache Spark Support the ongoing development and improvement of the data platform Ensure data remains securely hosted on MoD-owned infrastructure Collaborate with wider technical and delivery teams Troubleshoot and resolve technical issues across existing data pipelines Required Experience: Strong commercial experience ...

Data Analytics & Business Intelligence (BI) Experience: 5-10 years Location: United Kingdom - Glasgow, London Skills: Apache Airflow, AWS, Data Lake/Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural … work closely with System Analysts, Automation Engineers, and Data Architects to convert hundreds of legacy scripts into modern dbt models, orchestrate them via Astronomer (Airflow), and establish governed warehousing in Databricks. Requirements Required Skills & Technologies Cloud Platforms: Hands‐on engineering experience within the AWS ecosystem (S3, IAM, Glue ...

Data Analytics & Business Intelligence (BI) Experience 5-10 years Location United Kingdom - Glasgow, London Skills Apache Airflow, AWS, Data Lake/Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural … work closely with System Analysts, Automation Engineers, and Data Architects to convert hundreds of legacy scripts into modern dbt models, orchestrate them via Astronomer (Airflow), and establish governed warehousing in Databricks. Requirements Required Skills & Technologies Cloud Platforms: Hands-on engineering experience within the AWS ecosystem (S3, IAM, Glue ...

Description Job Description Software Engineer, Airflow Data Pipelines Make an Impact Where It Matters Most At Intelliforce, you build mission software with teams who move fast, communicate clearly, and own outcomes. In this role, you will engineer reliable data workflows that help turn complex data into usable results. … This role is a strong fit for an engineer who likes building systems that do real work every day. You will design and sustain Apache Airflow workflows, partner with data and platform teams on scalable automation patterns, and strengthen reliability across Linux-based, containerized environments. The work ...

Description Job Description Software Engineer, Airflow Data Pipelines Make an Impact Where It Matters Most At Intelliforce, you build mission software with teams who move fast, communicate clearly, and own outcomes. In this role, you will engineer reliable data workflows that help turn complex data into usable results. … This role is a strong fit for an engineer who likes building systems that do real work every day. You will design and sustain Apache Airflow workflows, partner with data and platform teams on scalable automation patterns, and strengthen reliability across Linux-based, containerized environments. The work ...

love the work!) In this key Software Engineer role, you'll be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow, keeping critical processes flowing as smoothly as a well-timed morning coffee. This role focuses on building reliable, scalable, and observable workflow orchestration … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

love the work!) In this key Software Engineer role, you'll be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow, keeping critical processes flowing as smoothly as a well-timed morning coffee. This role focuses on building reliable, scalable, and observable workflow orchestration … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

working technical solutions. Key Responsibilities: Build and maintain scalable applications using Python, SQL and PySpark Develop and support large-scale data processing pipelines using Apache Spark Use Apache Airflow for workflow orchestration and task scheduling Work with AWS EMR environments and review logs/output to troubleshoot … with stakeholders to translate customer requirements into technical functionality Follow engineering best practices across delivery team Technical Environment: Programming/Data: Python SQL PySpark Apache Spark Apache Airflow AWS/Cloud: AWS EMR S3 Glue CloudWatch IAM EC2 ECR DynamoDB Lake Formation Amazon Athena AI/Intelligent ...

processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex business … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries ...

Captivation Software is looking for a mid level software engineer who shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

Captivation Software is looking for a mid level software engineer who shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

deliver secure, reliable, and scalable services that support critical public sector operations. Key Responsibilities Design, develop, and maintain data pipelines and processing workflows using Apache Spark and Python Build, schedule, and monitor workflow orchestration using Apache Airflow Develop and maintain infrastructure-as-code using Terraform across … sprint planning, refinement, and other agile ceremonies Required Skills & Qualifications Experience with AWS services Experience with Terraform for infrastructure as code Experience with Apache Spark for data processing Experience with Apache Airflow for workflow orchestration Experience with Docker and containerisation Experience building and maintaining CI/ ...

Linux operating systems, including command-line administration and troubleshooting. Solid experience with SQL or data pipeline technologies, Experience with data processing frameworks such as Apache Airflow, Apache Flink or Google Dataflow, with a practical understanding of ETL pipeline design and the ability to diagnose issues across data …/IP networking principles, including the ability to diagnose connectivity issues and interpret network traffic. Familiarity with event streaming platforms such as Apache Kafka, including an understanding of producer/consumer patterns and how message queues are used to move data reliably between systems at scale. Familiarity with infrastructure ...

software development team Hands-on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands-on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … highly scalable data engineering solutions that power Moody’s next‐generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moody’s technical ...

software development team Hands-on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands-on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … support highly scalable data engineering solutions that power Moodys next-generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moodys technical teams ...

Lead the development of transformation workflows using dbt, with a strong emphasis on testing, documentation, and data quality. Architect and manage pipeline orchestration via Apache Airflow, ensuring reliability and observability at every stage. Monitor and optimise pipeline performance for low-latency, high-throughput use cases. Collaborate closely with … customer and marketing data at scale. Expert-level SQL and proven experience building robust transformation pipelines with dbt. Solid experience orchestrating workflows with Apache Airflow in production environments. Comfortable working across cloud platforms (AWS, GCP, or Azure) for data ingestion, storage, and processing. Proficiency in Python ...

London --------------------------------------- JOB DETAILS Role Title: Data Engineer (Python Enterprise Developer) Must have skills: -Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow -Preferred Domain: Energy trading Minimum number of relevant years of experience: 8+ Detailed Job Description: JD: • 8+ years of experience in python … communication skills with an ability to lead a team and keep them motivated. • Python, Postgres SQL, Azure Databricks, AWS(S3), Git, Azure DevOps CICD, Apache Airflow #J-18808-Ljbffr ...

articulate technical decisions to both engineering and non-technical stakeholders. NICE TO HAVE: C# or other strongly-typed backend language experience. Data pipeline experience (Apache Airflow, Apache Flink, Kafka, or similar). Workflow/orchestration engine experience (Temporal.io, Conductor, or similar). Figma familiarity - able to work ...

articulate technical decisions to both engineering and non-technical stakeholders. NICE TO HAVE: C# or other strongly-typed backend language experience. Data pipeline experience (Apache Airflow, Apache Flink, Kafka, or similar). Workflow/orchestration engine experience (Temporal.io, Conductor, or similar). Figma familiarity - able to work ...

country. Description Captivation Software is looking for a senior level software engineer who shall possess extensive expertise in dataflow design, data transport mechanisms, and Apache Spark based distributed processing. In this role, the Software Engineer shall be responsible for designing, implementing, and optimizing data ingress/egress pathways … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience with Distributed Big Data processing engines including Apache Spark ...

country. Description Captivation Software is looking for a senior level software engineer who shall possess extensive expertise in dataflow design, data transport mechanisms, and Apache Spark based distributed processing. In this role, the Software Engineer shall be responsible for designing, implementing, and optimizing data ingress/egress pathways … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience with Distributed Big Data processing engines including Apache Spark ...

manage healthcare data at scale. Data Pipelines & Cloud Infrastructure Build, monitor, and optimize large-scale healthcare data pipelines. Develop workflow orchestration solutions utilizing Apache Airflow and/or Google Cloud Composer. Improve scalability, performance, observability, reliability, and operational efficiency across healthcare data systems. Support cloud-native architecture … Experience with cloud platforms, preferably Google Cloud Platform. Experience building and supporting data pipelines. Experience working with workflow orchestration tools. Preferred Requirements Experience with Apache Airflow and/or Google Cloud Composer. Experience with healthcare integration engines such as Mirth Connect, Rhapsody, Cloverleaf, or similar platforms. Experience working ...