1 to 25 of 289 Apache Airflow Jobs in London

Consultant to support the implementation and deployment of Astronomer (Astro) within a leading asset management organisation. Job Responsibilities/Objectives Astronomer is built around Apache Airflow and is used to develop, schedule, automate and manage complex data pipelines. The organisation is currently implementing the platform and requires … ownership of the implementation and providing training and support to upskill the wider team. Experience with Astronomer/Astro is essential, alongside knowledge of Apache Airflow, dbt and Snowflake. Key Responsibilities Lead the implementation, build and deployment of Astronomer (Astro). Configure and manage Apache Airflow ...

integration, and production support. Good understanding of ETL/ELT, data warehousing, data lakes, dimensional modelling, batch processing, and stream processing. Experience with Apache Beam, Kafka, REST APIs, Docker, Kubernetes, or GKE. Experience with data formats such as JSON, CSV, Avro, and Parquet. Working knowledge of Git, automated testing … observability, and release management. Ability to design solutions for scalability, resilience, security, performance, operability, and cost efficiency. Preferred Skills Experience with Cloud Composer or Apache Airflow, Dataproc or Spark, Dataform or dbt, and metadata-driven pipeline frameworks. Exposure to Terraform or another infrastructure-as-code tool. Knowledge ...

bring mission-critical software, analytics, and AI to life and is the company behind Astro, the industry-leading unified DataOps platform powered by Apache Airflow®. Astro accelerates building reliable data products that unlock insights, unleash AI value, and powers data-driven applications. Trusted by more than … this role: Astronomer is on a mission to make DataOps a first-class discipline in every modern data organization. As the driving force behind Apache Airflow, we're powering mission-critical pipelines at scale for companies like Condé Nast, Rappi, Sonos, and more. As a Sales Engineer ...

Data Engineer — AWS, Python & Airflow Rate: £550 per day, outside IR35 Location: Remote Duration: Initial 6-month contract, with a further 12-month extension planned Clearance: Active SC clearance, used within the last 12 months Nationality: Sole British nationals only; dual citizenship is not permitted The opportunity … urgently looking for an experienced Data Engineer with strong AWS, Python and Apache Airflow skills to join a programme on an initial six-month contract. You'll help build and maintain reliable data pipelines, develop cloud-based data solutions and make data available for downstream analytics and programme ...

Build repeatable evaluations, investigate failure cases and use the results to guide model and pipeline improvements. Orchestrate data and model workflows. Build and maintain Apache Airflow pipelines, with monitoring, retries and clear approaches to failure recovery. Deploy and operate AI services. Work with AWS services including Lambda … through unfamiliar problems. You’re comfortable discussing trade-offs, asking questions and taking responsibility for delivery. Useful additional experience LLMs or NLP, GPU workloads, Apache Airflow, AWS, infrastructure-as-code tools such as Pulumi, or financial-market data. You don’t need experience with every tool ...

expertise in writing complex, optimized SQL queries for data extraction and transformation. ETL/ELT: Hands-on experience building data pipelines using frameworks like Apache Spark, Apache Airflow, DBT, Fivetran, Matillion, or equivalent. Databases: Solid understanding of relational databases (e.g., PostgreSQL, MySQL) and experience with modern data ...

accountability mindset with ability to mentor and guide team members* (Desirable) Experience with streaming platforms such as Kafka or Kinesis* (Desirable) Experience with Airflow or other workflow orchestration tools* (Desirable) Experience with Docker and Kubernetes* (Desirable) Exposure to Data Lake and Lakehouse architecture* (Desirable) Experience in Agile/Scrum … development environments* (Desirable) Experience with Apache Kafka, Apache Airflow, Terraform, Databricks, Delta Lake, Snowflake, CI/CD pipelines, or Machine Learning data pipelines* (Desirable) Bachelor's or Master's degree in Computer Science, Engineering, or a related field* Language(s): English (fluent)## Job Description* Design, develop ...

Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

modern AI systems. Strong programming experience in Python. Data & Infrastructure Experience with some of the following technologies is highly desirable: Frameworks/Infrastructure Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Libraries Pandas Polars Engineering ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

knowledge of optimisation techniques for large‐scale data processing Strong proficiency in SQL and Python for handling complex data problems Hands‐on experience with Apache Spark (PySpark or Spark SQL) Experience with the Azure data stack Knowledge of workflow orchestration tools like Azure Data Factory or Apache Airflow … cost optimisation strategies for cloud data platforms Experience with data quality frameworks and implementation Experience with data visualisation tools like Power BI or Apache Superset Experience with other cloud data platforms like AWS, GCP or Oracle Experience with modern unified data platforms like Databricks or Microsoft Fabric Experience with ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

stores). Hands-on experience building, scaling, and shipping large-scale data platforms in production. Deep practical experience with distributed data processing frameworks (e.g., Apache Spark, Flink, Hadoop). Strong expertise in message brokers and event streaming platforms (e.g., Apache Kafka, Kinesis). End-to-end exposure … data pipeline lifecycle development, including extensive experience with workflow orchestration tools (e.g., Apache Airflow, Dagster). Hands-on expertise with cloud data warehouses (e.g., Snowflake, BigQuery, Redshift) and data lake architectures (e.g., Databricks, Delta Lake, Apache Iceberg). Advanced SQL skills and proficiency in Python. Strong background ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

experience designing and building scalable data pipelines (ETL/ELT) for batch and streaming use cases using modern technologies (e.g., Python, SQL, Spark, Flink, Airflow) and advanced knowledge on the workings of these. Hands-on experience in building and operating workloads Databricks. Hands-on experience creating data visualizations ...

Athena, Lambda, S3, Glue and Step Functions, but we're open to transferable experience solving similar problems. Experience with Snowflake, Spark, Scala or Airflow, or transferable experience with comparable technologies and solving similar data engineering challenges. Experience with Git, GitHub and CI/CD practices, including tools such ...

production pipelines Strong SQL and Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

operational issues. Understanding of platform security, resilience, scalability and availability. Experience working within Agile and/or DevOps delivery environments . Desirable: Helm, ArgoCD, Airflow, dbt, Bash, Ansible, Packer, Elasticsearch/OpenSearch, Prometheus, Grafana, ELK or Datadog. Experience working within public sector, utilities or other regulated environments is advantageous. ...

translate them into a practical, maintainable data solution Strong problem‐solving and analytical skills Orchestration and monitoring tools, e.g. Databricks Workflows, dbt or Airflow (desirable) Integrating SaaS platforms through APIs, e.g. Salesforce or Workday (desirable) Building data catalogues, dictionaries or certified datasets, e.g. Unity Catalog, Microsoft Purview or Power ...

adhere to the D.R.Y principle, minimising manual intervention and promote automation. We're looking for these essential skills Hands‐on experience with DBT and Apache Airflow, building scalable data transformations and DAGs to orchestrate complex, reliable data pipelines. Manage task dependencies, automated retries and error handling to ensure ...

Microsoft Fabric . Strong Python, SQL and PySpark development skills. Experience building scalable ETL/ELT pipelines and data integration solutions. Experience with DBT, Airflow, Azure Data Factory or Dataflow . Knowledge of data modelling, data warehousing and Lakehouse architectures . Experience with data quality, governance and CI/ ...