126 to 150 of 928 Permanent Apache Airflow Jobs

implementing agentic guardrails. Data Engineering (30%): Practical experience implementing components of data pipelines, including real-time streaming tools (AWS Kinesis, Kafka), data orchestration (dbt, Airflow), and managing Vector Databases for RAG architectures. Observability & Cost Management: SRE/Platform experience with the practical application of real-time monitoring and cloud ...

Kubernetes, Docker, and infrastructure-as-code tooling Data Platforms: Snowflake, Spark, SQL, S3; advanced data engineering workflows and high-scale data pipelines ETL & Orchestration: Airflow or AWS Glue Streaming: Kafka or Kinesis for real-time ingestion and system integrations Observability: Enterprise-level telemetry systems including DataDog, Grafana, and CloudWatch ...

Kubernetes, Docker, and infrastructure-as-code tooling Data Platforms: Snowflake, Spark, SQL, S3; advanced data engineering workflows and high-scale data pipelines ETL & Orchestration: Airflow or AWS Glue Streaming: Kafka or Kinesis for real-time ingestion and system integrations Observability: Enterprise-level telemetry systems including DataDog, Grafana, and CloudWatch ...

required . Proficiency in at least one object-oriented language. Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto. Experience with AWS, GCP, or Looker (advantageous but not essential). Solid background in data modelling, including the design, implementation and optimisation ...

required . Proficiency in at least one object-oriented language. Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto. Experience with AWS, GCP, or Looker (advantageous but not essential). Solid background in data modelling, including the design, implementation and optimisation ...

wider stakeholders. Technical Environment Experience with some of the following technologies would be beneficial: Python SQL Cloud platforms – AWS, Azure or GCP Databricks Apache Spark/PySpark Kafka/event-driven architectures Airflow dbt Git/GitHub CI/CD Infrastructure as Code Modern data platforms and distributed ...

Data Engineer within GCP environments Hands-on experience with BigQuery, Dataflow and Spanner Strong programming capability in Python, Java or Scala Experience with Apache Beam/Dataflow and distributed processing frameworks Experience designing and managing ETL/ELT pipelines Solid understanding of data modelling and database design Experience with … workflow orchestration tools such as Apache Airflow Strong understanding of data governance, security and scalability Desirable Skills for the GCP Data Engineer: Experience with Pub/Sub, Cloud Composer or Cloud Data Fusion Exposure to CI/CD, DevOps and Infrastructure as Code Experience with real-time streaming ...

operating ETL/ELT pipelines and large‐scale data architectures. Hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

find to improve model and system performance. Build reliable ML infrastructure Develop and maintain data and machine learning workflows using technologies such as Apache Airflow. Help improve CI/CD, automated testing, monitoring and reproducibility across our ML stack. Engineer on AWS Build and improve cloud infrastructure using services … expect you to know everything: PyTorch, TensorFlow or JAX Hugging Face LLM APIs and open-source models RAG and vector databases Apache Airflow AWS S3 ECS/EKS Redshift Docker Kubernetes GitHub Actions Pulumi or Terraform SQL Model monitoring and observability Distributed processing Financial markets, economics or commodities ...

production problems and taking responsibility for critical infrastructure. Startup Mindset: Proactive, resourceful and comfortable taking ownership in a fast-moving engineering environment. Bonus : with Apache Airflow, real-time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial-market systems. Why Permutable Real Ownership: Take significant ...

practice and mentoring less experienced engineers. Strong stakeholder management and communication skills. Experience working in a regulated environment. Required Tech Stack Python DBT Snowflake Airflow Azure Terraform Nice to Have Experience with Kubernetes/AKS Infrastructure and platform modernisation projects Strong CI/CD and DevOps practices #J ...

offer the following skills & experience. Strong experience in Python, PySpark/Spark, Databricks, Azure Data Factory, SQL DB, Synapse, Stream Analytics, Glue, Airflow, Kinesis, Redshift, SonarQube, PyTest. Experience designing, building and supporting scalable data pipelines and data products in cloud environments. Experience developing and maintaining REST APIs using frameworks ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

debugging, the advanced stuff that separates architects from diagram-drawers. Lakehouse fluency — Medallion architecture, Delta Lake, Unity Catalog, governance, orchestration (Workflows, DLT/Lakeflow, Airflow, ADF); Microsoft Fabric exposure a bonus. Battle-tested in migrations — you've led large transformation programs and have the scars and success stories ...

maintaining data pipelines at scale Experience with a modern data stack (e.g. Snowflake, dbt, Fivetran, Debezium or similar) Experience with orchestration tools such as Airflow, Dagster or Prefect Proficiency in Python for building production data workflows Familiarity with data warehousing concepts and modelling patterns Experience working with cloud platforms ...

data technology such as BigQuery, Snowflake, Redshift, Hive, Spark, Vertica, or StarRocks; Experience with ETL/ELT tools or orchestration frameworks such as Airflow, dbt, or internal frameworks; Proficiency in at least one programming language for data and services, such as Python, Java, Golang, Kotlin, or JavaScript/TypeScript ...

using: Kafka as our message transport Flink for (near) real time processing Datahub as our catalog BigQuery as our warehouse Airflow (GCP Composer) for scheduling DBT for data transformation Montecarlo for monitoring The platform here encompasses the end-to-end, first for real time/streaming use cases ...

mobile) Familiarity with cloud platforms (AWS, GCP, or Azure), containerization (Docker), and orchestration (ECS, Kubernetes) Proven experience in building and maintaining data pipelines (e.g., Airflow) Strong understanding of the agile development process and CI/CD pipelines and tools (e.g., Github Actions, Jenkins) Excellent communication skills, capable of presenting ...

. Knowledge of regulatory reporting and treasury operations in retail bankingExposure to Python, Go or similar languages. Experience working with orchestration frameworks such as Airflow/LuigiHave previously used dbt, dataform or similar tooling. Used to AGILE ways of working (Kanban, Scrum)The Interview Process: Our interview process involves ...

. Knowledge of regulatory reporting and treasury operations in retail bankingExposure to Python, Go or similar languages. Experience working with orchestration frameworks such as Airflow/LuigiHave previously used dbt, dataform or similar tooling. Used to AGILE ways of working (Kanban, Scrum)The Interview Process: Our interview process involves ...

Backstage experience - software catalog configuration, scaffolder templates, or plugin development* Familiarity with data platform concepts - medallion architecture, lakehouse patterns, or orchestration tools such as Airflow or dbt* Exposure managing Databricks* Practical experience with AI coding assistants (Claude Code, GitHub Copilot, Cursor, or similar) in a professional engineering context* Experience ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

POLYGRAPH LEVEL IS REQUIRED We have an immediate need for a Software Engineer with extensive expertise in dataflow design, data transport mechanisms, and Apache Spark distributed processing. In this high-impact role, you will be responsible for designing, implementing, and optimizing end-to-end data ingress and egress pathways. … work experience, education, and/or skill level, among other things. , Required Skills Distributed Processing: Extensive experience with distributed Big Data processing engines, specifically Apache Spark . Software Development: Recent, hands-on software development experience using both Python and Java . Workflow Orchestration: Practical experience using Apache Airflow ...

POLYGRAPH LEVEL IS REQUIRED We have an immediate need for a Software Engineer with extensive expertise in dataflow design, data transport mechanisms, and Apache Spark distributed processing. In this high-impact role, you will be responsible for designing, implementing, and optimizing end-to-end data ingress and egress pathways. … work experience, education, and/or skill level, among other things. , Required Skills Distributed Processing: Extensive experience with distributed Big Data processing engines, specifically Apache Spark . Software Development: Recent, hands-on software development experience using both Python and Java . Workflow Orchestration: Practical experience using Apache Airflow ...