1 to 25 of 298 Remote/Hybrid Permanent Apache Airflow Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex business … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries ...

Captivation Software is looking for a mid level software engineer who shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

Captivation Software is looking for a mid level software engineer who shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

Lead the development of transformation workflows using dbt, with a strong emphasis on testing, documentation, and data quality. Architect and manage pipeline orchestration via Apache Airflow, ensuring reliability and observability at every stage. Monitor and optimise pipeline performance for low-latency, high-throughput use cases. Collaborate closely with … customer and marketing data at scale. Expert-level SQL and proven experience building robust transformation pipelines with dbt. Solid experience orchestrating workflows with Apache Airflow in production environments. Comfortable working across cloud platforms (AWS, GCP, or Azure) for data ingestion, storage, and processing. Proficiency in Python ...

Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

Modern data platforms: Databricks, Snowflake and cloud-native data services Data architecture: Lakehouse, medallion, data mesh and domain-oriented patterns Data processing: Apache Spark, SQL and distributed processing technologies Data storage: Delta Lake, Apache Iceberg and open table formats Data pipelines: Enterprise-scale ETL/ELT, batch … streaming patterns Transformation and quality: dbt, Great Expectations and equivalent tooling Orchestration: Apache Airflow and cloud-native orchestration services Event streaming: Kafka and cloud-native event-streaming technologies Data modelling: Dimensional, analytical and domain-oriented modelling Data governance: Catalogue, metadata, lineage, quality, security, privacy and access controls Data ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

Synapse/DatabricksAWS Glue/Redshift/EMRGoogle BigQuery/DataflowEnsure systems are secure, scalable, and cost-efficientData Pipeline & Workflow ManagementDevelop orchestration workflows (e.g., Airflow, Prefect, Azure Data Factory)Monitor and troubleshoot pipelines to ensure high availability and performanceImplement robust error handling, logging, and alertingData Quality & GovernanceEnsure data quality ...

optimisation and administration Strong programming experience with Python, Go and/or SQL Experience using dbt for data modelling and transformation Experience working with Apache Airflow or similar orchestration tooling Proven experience designing and building production systems on GCP and/or AWS Strong understanding of event-driven … distributed systems Experience deploying and supporting batch and real-time machine learning workloads Experience working with distributed processing technologies such as Spark or Apache Beam Strong understanding of data quality principles and engineering best practices Experience optimising database performance, troubleshooting issues and maintaining production data systems Ability to learn ...

optimisation and administration Strong programming experience with Python, Go and/or SQL Experience using dbt for data modelling and transformation Experience working with Apache Airflow or similar orchestration tooling Proven experience designing and building production systems on GCP and/or AWS Strong understanding of event-driven … distributed systems Experience deploying and supporting batch and real-time machine learning workloads Experience working with distributed processing technologies such as Spark or Apache Beam Strong understanding of data quality principles and engineering best practices Experience optimising database performance, troubleshooting issues and maintaining production data systems Ability to learn ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure ...

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure ...

analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

data pipelines, orchestration, and automation; Solid software engineering fundamentals, including version control, testing, and CI/CD; Experience with workflow orchestration frameworks such as Apache Airflow; Experience with cloud-native data platforms, preferably AWS or Azure; Experience with enterprise data warehouses, preferably Snowflake; Experience with performance tuning, cost ...

. Strong technical knowledge of SQL and Python . Experience with modern data technologies and tooling such as Microsoft Fabric, Power BI, Snowflake, dbt, Airflow, Databricks or Looker . Experience designing data pipelines, ETL/ELT processes and modern data models. Practical understanding of modern data architecture approaches, including ...

Experience with cloud data services and technologies, ideally within the GCP ecosystem, such as Dataflow, Pub/Sub, BigQuery or Vertex AI. Experience with Airflow environment management and Docker. An interest in contributing to the AI roadmap and experimenting with AI tooling, including coding agents, within the development lifecycle. ...

large- scale batch processing. Proven ability to deploy and maintain data infrastructure using Infrastructure as Code principles and tools such as Terraform. Experience with Airflow environment management and Docker. Experience contributing to an AI roadmap and evaluating AI tooling, including coding agents, for use within the development lifecycle. ...