51 to 75 of 237 Apache Airflow Jobs in the UK

. ML workflow orchestration and serving: Operate the systems researchers use to launch experiments, data jobs, and production endpoints — workflow engines (e.g., Kubeflow Pipelines, Airflow), GPU schedulers (e.g., Volcano, Slurm), experiment trackers (e.g., MLflow, Weights & Biases), and managed‐inference platforms (e.g., Modal, Triton) — and maintain a launcher ...

within multi-disciplinary teams, delivering modern data platforms into large organisations. Previous practical experience of using both SQL and task orchestration software (eg. dbt, Airflow/Prefect/Dagster or equivalent) to develop production-grade pipelines that bring data to life Strong Snowflake experience is preferred, and Databricks experience ...

preparation specifically geared toward AI model integration Technical Fluency: A strong conceptual understanding of modern data ecosystems, data warehousing, pipeline orchestration tools (e.g., dbt, Airflow), and data governance standards Cross-Functional Communication: Outstanding stakeholder management skills, with a demonstrated ability to foster collaboration between technical developers and non-technical ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

Backstage experience - software catalog configuration, scaffolder templates, or plugin development* Familiarity with data platform concepts - medallion architecture, lakehouse patterns, or orchestration tools such as Airflow or dbt* Exposure managing Databricks* Practical experience with AI coding assistants (Claude Code, GitHub Copilot, Cursor, or similar) in a professional engineering context* Experience ...

environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high … commercial experience developing data engineering solutions using Python . Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake. Strong experience with Apache Spark/PySpark . Proven experience building scalable ETL/ELT pipelines. Strong SQL and data modelling skills. Experience designing modern Lakehouse architectures. Experience ...

demand generation strategies. Key Goals & Responsibilities: Data Integration and Automation: Leverage your expertise to build robust automation processes for seamless data integration. Utilize Apache Airflow to create and manage complex data workflows, ensuring efficient and reliable data processing and integration tasks. Take advantage of MS Power Automate … Requirements/Qualifications: Passionate Data Enthusiast: A genuine and unwavering passion for the world of data engineering, analytics, and data science. Proficiency in Python (Airflow, Pandas, NumPy) is essential for both data science and data engineering tasks. TensorFlow, SciPy, PySpark, Matplotlib, Seaborn experience is a significant plus. Data Modeling ...

others. Active SC Clearance, or eligibility to obtain. Desirable: SAS Viya for data analytics. Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). Denodo for data virtualisation. Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). ALM Tooling (Jira, Confluence ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

communication skills. Highly Desirable: Experience with Spark/PySpark. Familiarity with Azure Data Lake, Snowflake, or similar platforms. Knowledge of orchestration tools like Airflow or Azure Data Factory. Exposure to financial services environments. Experience supporting large-scale data transformation or migration programmes. What We're Looking For: A senior ...

software engineering skills in Python, with experience building scalable services, APIs, and production-quality code Experience with modern ML tooling and platforms (e.g.Databricks,MLflow, Airflow, Kubeflow, SageMaker, Vertex AI) Familiarity with CI/CD pipelines and infrastructure-as-code (e.g.Terraform, CloudFormation) Experience building robust, maintainable, and testable ML pipelines ...

technologies and practices such as Docker, Terraform, CI/CD pipelines and infrastructure‐as‐code. Familiarity with MLOps tools and practices such as MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark ...

implementation of modern data architectures and concepts such as cloud services (e.g., AWS, Azure, GCP), real-time data distribution (e.g., Kafka, Kinesis, DataFlow, Airflow), NoSQL (e.g., MongoDB, DynamoDB, HBase, CosmosDB), GraphQL and modern data warehouse tools including Snowflake and DataBricks* Ability to think strategically and relate architectural decisions ...

implementation of modern data architectures and concepts such as cloud services (e.g., AWS, Azure, GCP), real-time data distribution (e.g., Kafka, Kinesis, DataFlow, Airflow), NoSQL (e.g., MongoDB, DynamoDB, HBase, CosmosDB), GraphQL and modern data warehouse tools including Snowflake and DataBricks Ability to think strategically and relate architectural decisions ...

conscious: optimised warehouse spend, storage strategies, or job efficiency, and treat compute as a resource to manage, not ignore Platform fluency: worked across orchestration (Airflow, Dagster), warehouses (Snowflake, BigQuery, Databricks), ingestion (Fivetran, Airbyte, custom), and transformation (dbt, Spark) AI Native: experienced in agentic coding workflows (context, harness, loop ...

common CI/CD stack (e.g., Jenkins, CircleCI, GitLab, Azure DevOps) Experience in workflow orchestration with tools such as Argo Workflow, Airflow, and scientific workflow tools such as Nextflow, Snakemake, VisTrails, or Cromwell Experience with application performance tuning and optimization, including in parallel and distributed computing paradigms and communication ...

Excellent verbal and written communication skills BS or MS degree in Computer Science or equivalent Experience in distributed computing frameworks like Hive/Hadoop, Apache Spark Experience in developing Finance or HR related applications Experience with cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic Cache … Lambda Working experience with Terraform Experience in creating workflows for Apache Airflow What the job involves As a senior technical lead, you consider yourself a platform services developer and develop innovative solutions using modern software frameworks and paradigms You are self‐driven and a problem solver, with ...

Data Engineer to design, build, and maintain scalable data pipelines powering Moody’s next-generation digital content platform. You will work with DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS, driving platform optimizations and cross-functional collaboration. In this role, you’ll collaborate with product owners ...

software and site reliability engineering. Preferred experience with distributed ML frameworks such as Horovod or TensorFlow Distributed, familiarity with data engineering pipelines such as Apache Airflow or Spark, and knowledge of chaos engineering tools and compliance frameworks such as GDPR, SOC 2, or ISO 27001. Benefits Relocation benefits ...

Strong cloud data platform expertise, including Google Cloud Platform (GCP). Hands-on experience with BigQuery, Dataflow, Datastream, Data Fusion, Cloud Functions, Dataproc, and Airflow/Composer. Advanced SQL and Python skills. Expertise in data lakes, data warehouses, and modern data architectures. Strong understanding of data governance, lineage ...

tools. Experience building and optimising scalable data pipelines and data warehouses. Knowledge of AWS technologies such as Redshift, S3 and Athena. Experience with Airflow, Fivetran or similar orchestration and integration tools. Strong Linux/Unix and Bash scripting skills. A solid understanding of data quality, governance, lineage and monitoring. ...

database querying languages Experience in large‐scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Proficiency in automation, continuous delivery methods, and all aspects of the Software Development Life Cycle Advanced understanding of agile methodologies, CI/CD, application ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

Practical experience with Azure (Azure OpenAI, Azure AI Foundry, AI Search) and familiarity with AWS or GCP Proficiency with containerized deployment, MLOps tooling (MLFlow, Airflow), and infrastructure‐as‐code practices Bachelor’s or Master’s degree in Computer Science, Data Science, or related field Nice to have Experience with ...

database querying languages Experience in large‐scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Proficiency in automation, continuous delivery methods, and all aspects of the Software Development Life Cycle Advanced understanding of agile methodologies, CI/CD, application ...