1 to 25 of 45 Remote/Hybrid Permanent Apache Airflow Jobs

will be undertaking a monumental Transformation to unlock organisational pace and Data agility, moving to a scalable Cloud Data architecture centred around AWS Glue (Apache Spark) & Apache Airflow, with a key focus on re-engineering numerous complex Data pipelines to the new framework by leveraging Agentic … through to validated delivery Architect and build the Claude Code agentic workflow: Parse existing data movement pipelines and logic Generate equivalent PySpark transformations Produce Airflow DAGs Operate, iterate, and scale across the full data pipeline estate Ensure rigorous creation and management of the PySpark data processing layer within ...

role We are seeking a high‐skilled Senior Software Engineer with deep expertise in API Development using Spring Boot while also possessing knowledge of Apache Spark‐based Big Data pipelines using Airflow. This hybrid position bridges software engineering and data engineering, requiring the ability to design, build, and maintain … teams to architect and manage robust, production‐grade data products that power critical analytics and measurement capabilities. You will work with technologies such as Apache Spark, Apache Airflow, Trino, Druid, Spring Boot, StarRocks, and Looker to deliver reliable, high‐performance solutions. The ideal candidate is a proactive ...

tech skills required include • Great SQL • Experience in complex query optimization • Strong Python programming skills with experience in data processing libraries (pandas, NumPy, Apache Spark) • Hands-on experience building and maintaining data ingestion pipelines • Proven track record of optimising queries, code, and system performance • Experience with open-source data … processing frameworks (Apache Spark, Apache Kafka, Apache Airflow) If you are interested in this Data engineering role, please send your latest CV to ross@virtuetech.io Data Engineers x 2 | Open-source data processing, SQL, Python | Energy trading | Up to £85,000 + Bonus + Benefits | Hybrid ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality of coding ...

looking for: · 3+ years of professional Python development experience as a Data Platform Engineer, using AWS. · Experience developing and deploying Serverless technologies, provisioning Apache Airflow, and using infrastructure as code such as AWS CloudFormation or Terraform. · Takes a collaborative approach to problem solving and breaking down complex problems … with software engineering requirements and specifications. · Good interpersonal skills, positive can-do attitude and willing to help other members of the team. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Exposure to using AI tools to enhance productivity and quality of coding ...

maintain scalable and robust data pipelines using SQL and Python, and leveraging any of the following: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

data science, or a hybrid role Strong programming skills in Python and SQL Experience building data pipelines and working with large datasets (e.g., Spark, Airflow, or similar tools) Hands-on experience with cloud platforms (AWS, Azure, or GCP) Familiarity with machine learning frameworks (e.g., scikit-learn, TensorFlow, PyTorch) Experience ...

Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills with ...

hands-on and at a strategic level Experience working in regulated environments (financial services preferred) Nice to have: Experience with orchestration tools such as Airflow or Azure Data Factory Exposure to dbt or similar transformation frameworks Experience supporting ML pipelines and lifecycle management Familiarity with Azure DevOps, Git-based ...

capabilitie Requirements: 5+ years of experience in data engineering within production environments Strong Python and SQL skills Experience with workflow orchestration tools such as Airflow, Dagster, Prefect, or similar Hands-on experience with transformation frameworks such as dbt or Spark Experience with cloud data warehouses such as Snowflake, BigQuery ...

hands-on and at a strategic level Experience working in regulated environments (financial services preferred) Nice to have: Experience with orchestration tools such as Airflow or Azure Data Factory Exposure to dbt or similar transformation frameworks Experience supporting ML pipelines and lifecycle management Familiarity with Azure DevOps, Git-based ...

hands-on and at a strategic level Experience working in regulated environments (financial services preferred) Nice to have: Experience with orchestration tools such as Airflow or Azure Data Factory Exposure to dbt or similar transformation frameworks Experience supporting ML pipelines and lifecycle management Familiarity with Azure DevOps, Git-based ...

Building Extract Transform and Load (ETL) pipeline (streaming/batch) using MPP frameworks (Spark, Beam, or other) Experience in orchestrating complex pipelines using Airflow, Dagster or other Proficiency in one or more programming languages: Python, Java, GO or other Experience with AWS or GCP and their products ...

Experience using Git in a team environment Hands-on experience using AI coding tools (e.g., Copilot, Cursor, Claude, Codex) Nice to have: Experience with Airflow, Dagster or other orchestration tools Experience with data lakes (Synapse, Snowflake) Experience with Docker Experience with streaming technologies (Kafka, RabbitMQ) Experience with Spark ...

Glue or EKS Hands on experience with modern analytics stacks such as Snowflake, dbt and Iceberg Familiarity with orchestration tools such as Dagster or Airflow Strong understanding of data modelling concepts and analytics best practice A software engineering mindset, including Git, testing and CI/CD workflows What They ...

technical solutions meet business needs. Experience with data ingestion tools, like Fivetran. Advantageous Exposure to deploying applications with Kubernetes. Experience with Data Orchestrator tools (Airflow, Prefect, etc.). Experience with Data Observability tools (Montecarlo, Great Expectations, etc.). Experience with Data Catalog tools (Amundsen, OpenMetadata, etc.). Benefits Winning ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

technical trade‐offs, backed by a previous background as an experienced software engineer. Experience with data platform and ETL/orchestration technologies such as Apache Airflow, Spark, AWS Glue, and Snowflake. Experience with or strong advocacy for AI developer tooling such as Claude Code, Copilot, Windsurf, and Gemini. ...

technical trade-offs, backed by a previous background as an experienced software engineer. Experience with data platform and ETL/orchestration technologies such as Apache Airflow, Spark, AWS Glue, and Snowflake. Experience with (or strong advocacy for) AI developer tooling such as Claude Code, Copilot, Windsurf, and Gemini. ...

Experience working within cloud environments (AWS or Azure) Strong SQL skills and experience working with large-scale datasets Familiarity with orchestration tools such as Airflow or similar Good understanding of data modelling and distributed data processing concepts Comfortable working across multiple client environments and stakeholders 🎁 What’s on Offer ...

Experience with CI/CD, microservices, TDD/BDD. Knowledge of secure system design. Desirable (AI & Data) Vector DBs: Milvus, Weaviate, Pinecone Data orchestration: Airflow, Prefect, Dagster Search: ElasticSearch/OpenSearch AI frameworks: LangChain, Hugging Face MLOps: MLflow, Kubeflow If you are looking for an opportunity to work ...

/ELT development, orchestration frameworks, cloud-based data platforms, and data quality management. You will be comfortable working with technologies such as dbt, Airflow, PostgreSQL, cloud warehouses or lakehouses, and modern data architecture patterns. Most importantly, you'll be someone who enjoys ownership, takes pride in building reliable solutions ...

Kubernetes and modern containerised deployment patterns Strong experience with CI/CD pipelines, infrastructure-as-code and workflow orchestration Experience with tools such as Airflow or similar platform and orchestration technologies Good understanding of model observability, data quality, feature pipelines, lineage and reproducibility Experience designing scalable infrastructure ...

organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design, Data Governance ...