151 to 175 of 947 Apache Airflow Jobs

using: Kafka as our message transport Flink for (near) real time processing Datahub as our catalog BigQuery as our warehouse Airflow (GCP Composer) for scheduling DBT for data transformation Montecarlo for monitoring The platform here encompasses the end-to-end, first for real time/streaming use cases ...

mobile) Familiarity with cloud platforms (AWS, GCP, or Azure), containerization (Docker), and orchestration (ECS, Kubernetes) Proven experience in building and maintaining data pipelines (e.g., Airflow) Strong understanding of the agile development process and CI/CD pipelines and tools (e.g., Github Actions, Jenkins) Excellent communication skills, capable of presenting ...

. Knowledge of regulatory reporting and treasury operations in retail bankingExposure to Python, Go or similar languages. Experience working with orchestration frameworks such as Airflow/LuigiHave previously used dbt, dataform or similar tooling. Used to AGILE ways of working (Kanban, Scrum)The Interview Process: Our interview process involves ...

. Knowledge of regulatory reporting and treasury operations in retail bankingExposure to Python, Go or similar languages. Experience working with orchestration frameworks such as Airflow/LuigiHave previously used dbt, dataform or similar tooling. Used to AGILE ways of working (Kanban, Scrum)The Interview Process: Our interview process involves ...

Backstage experience - software catalog configuration, scaffolder templates, or plugin development* Familiarity with data platform concepts - medallion architecture, lakehouse patterns, or orchestration tools such as Airflow or dbt* Exposure managing Databricks* Practical experience with AI coding assistants (Claude Code, GitHub Copilot, Cursor, or similar) in a professional engineering context* Experience ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

POLYGRAPH LEVEL IS REQUIRED We have an immediate need for a Software Engineer with extensive expertise in dataflow design, data transport mechanisms, and Apache Spark distributed processing. In this high-impact role, you will be responsible for designing, implementing, and optimizing end-to-end data ingress and egress pathways. … work experience, education, and/or skill level, among other things. , Required Skills Distributed Processing: Extensive experience with distributed Big Data processing engines, specifically Apache Spark . Software Development: Recent, hands-on software development experience using both Python and Java . Workflow Orchestration: Practical experience using Apache Airflow ...

POLYGRAPH LEVEL IS REQUIRED We have an immediate need for a Software Engineer with extensive expertise in dataflow design, data transport mechanisms, and Apache Spark distributed processing. In this high-impact role, you will be responsible for designing, implementing, and optimizing end-to-end data ingress and egress pathways. … work experience, education, and/or skill level, among other things. , Required Skills Distributed Processing: Extensive experience with distributed Big Data processing engines, specifically Apache Spark . Software Development: Recent, hands-on software development experience using both Python and Java . Workflow Orchestration: Practical experience using Apache Airflow ...

Configuration Management (CM) tools and Web Services implementation. The Software Engineer shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

Configuration Management (CM) tools and Web Services implementation. The Software Engineer shall be responsible for creating and maintaining data workflows and automation pipelines using Apache Airflow. This role focuses on building reliable, scalable, and observable workflow orchestration solutions that support data engineering, analytics, and operational use cases. The engineer … Linux CLI and Linux tools Experience developing Bash scripts to automate manual processes Recent software development experience using Python and Java Experience using Apache Airflow (DAG design, scheduling, operators, sensors) to orchestrate, schedule, and monitor complex workflows Experience using Distributed Big Data processing engines including Apache Spark ...

tooling, automated testing and reliable deployment processes. Startup Mindset: Proactive, resourceful and comfortable taking responsibility in a small, fast‐moving engineering team. Experience with Apache Airflow, Kubernetes, real‐time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial‐market systems would be useful ...

code Requirements: Must have strong, commercial AWS (contract) Data Engineering experience Must bring solid knowledge of data warehousing and modelling Experience with Airflow and infrastructure as a code (Terraform, CloudFormation) would be beneficial Must have experience with Redshift Nice to have: Experience with dbt/lake formation ...

code Requirements: - Must have strong, commercial AWS (contract) Data Engineering experience - Must bring solid knowledge of data warehousing and modelling - Experience with Airflow and infrastructure as a code (Terraform, CloudFormation) would be beneficial - Must have experience with Redshift Nice to have: - Experience with dbt/lake formation ...

preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers and semantic models Data Migration & Mapping Demonstrated ...

tools. Experience building and optimising scalable data pipelines and data warehouses. Knowledge of AWS technologies such as Redshift, S3 and Athena. Experience with Airflow, Fivetran or similar orchestration and integration tools. Strong Linux/Unix and Bash scripting skills. A solid understanding of data quality, governance, lineage and monitoring. ...

Arrow, plus an analytical engine, e.g. DuckDB, or a warehouse such as Snowflake. Orchestrated batch pipelines you've operated, not just written: Dagster or Airflow, S3-based data flows, and a habit of testing outputs for correctness rather than only for exceptions. Applied ML in production: embeddings and semantic ...

database querying languages Experience in large‐scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Demonstrated experience leading effective use of enterprise‐authorized AI‐assist­ed software development tools within the work environment (e.g., for coding, code review, test acceleration ...

systems, inventory extraction from application metadata tables (reporting opening). Columnar and lake formats (Parquet or similar), large object stores, orchestration platforms (Airflow or similar). Working-level knowledge of graph databases; data contracts and data quality frameworks; catalogue platforms (DataHub or similar) on the ingestion side. ...

data modelling Databricks as our datalake platform Kubernetes for data services Orchestra for task orchestration Terraform for infrastructure Lightdash for data applications Airflow and Astronomer for job scheduling and tracking Circle CI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Spark for data ...

influence technical and non‐technical stakeholders. Desirable Infrastructure‐as‐code (Terraform), Kubernetes, or Go/TypeScript experience. Experience in regulated industries. Familiarity with dbt, Airflow and event‐driven architectures. Admiral: Where You Can We take pride in being a diverse and inclusive business. It's a place where ...

experience building ML infrastructure, including model serving, training pipelines, feature stores, embeddings, or ML observability. Experience with modern data platforms such as BigQuery, Airflow, Spark, Beam/Dataflow, or streaming pipelines. Practical experience building production systems with LLMs or generative AI, including orchestration, provider APIs, observability, and performance optimization. ...

experience building ML infrastructure, including model serving, training pipelines, feature stores, embeddings, or ML observability. Experience with modern data platforms such as BigQuery, Airflow, Spark, Beam/Dataflow, or streaming pipelines. Practical experience building production systems with LLMs or generative AI, including orchestration, provider APIs, observability, and performance optimization. ...

experience building ML infrastructure, including model serving, training pipelines, feature stores, embeddings, or ML observability. Experience with modern data platforms such as BigQuery, Airflow, Spark, Beam/Dataflow, or streaming pipelines. Practical experience building production systems with LLMs or generative AI, including orchestration, provider APIs, observability, and performance optimization. ...

experience building ML infrastructure, including model serving, training pipelines, feature stores, embeddings, or ML observability. Experience with modern data platforms such as BigQuery, Airflow, Spark, Beam/Dataflow, or streaming pipelines. Practical experience building production systems with LLMs or generative AI, including orchestration, provider APIs, observability, and performance optimization. ...

experience building ML infrastructure, including model serving, training pipelines, feature stores, embeddings, or ML observability. Experience with modern data platforms such as BigQuery, Airflow, Spark, Beam/Dataflow, or streaming pipelines. Practical experience building production systems with LLMs or generative AI, including orchestration, provider APIs, observability, and performance optimization. ...