1 to 25 of 68 Apache Airflow Jobs in London

high-performance, batch and real-time data pipelines employing advanced streaming technologies like Kafka, and Flink. Utilise workflow orchestration tools such as Dataflow and Airflow Database and Storage Optimization : Optimize and manage a broad array of database technologies, from traditional relational databases (e.g., PostgreSQL, MySQL) to modern NoSQL solutions … e.g., MongoDB, Cassandra). Focus on strategies that enhance data accessibility, integrity, and performance Big Data Processing & Analytics : Utilise big data frameworks such as Apache Spark and Apache Flink to address challenges associated with large-scale data processing and analysis. These technologies are crucial for managing vast datasets ...

cloud environments, with hands-on experience in tools such as: Data transformation: DBT (Data Build Tool) Data warehousing: BigQuery or similar Workflow orchestration: Airflow Infrastructure as Code: Terraform Cloud Platforms: Google Cloud Platform (GCP) or Amazon Web Services (AWS) Strong software engineering fundamentals, including proficiency in Python ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

experience designing and building modern data platforms in AWS. ETL/Orchestration Expertise: Expertise in ETL/ELT design and data orchestration, specifically with Apache Airflow. SQL Mastery: Strong SQL skills with significant experience in query tuning and performance optimisation. Programming Proficiency: Proficiency in Python and Bash (for data ...

solving capabilities. · Excellent communication and documentation skills. · Ability to work independently and lead cross-functional teams. · Adaptability to rapidly evolving technological environments PREFERRED SKILLS · Airflow, dbt, Spark, Kafka, RabbitMQ, Redis. · Git, CI/CD pipelines. · Experience with data warehousing solutions such as Snowflake, Redshift, BigQuery, or Synapse. · Exposure ...

cost-efficient data pipelines (batch + streaming). Deep understanding of dimensional modeling, data warehouses, and industry best practices. Proficient with orchestration tools (Dagster, Airflow, Step Functions). Hands-on experience building automation and CI/CD workflows for data platforms. Practical experience applying Generative AI or LLM-based ...

Data Engineering (not building pipelines but designing and building the framework) DevOps MLOps/LLMOps Often work with the following technologies : Azure, AWS, GCP Airflow, dbt, Databricks, Snowflake, etc. GitHub, Azure DevOps and related developer tooling and CI/CD platforms, Terraform or other Infra-as-Code MLflow, AzureML ...

using concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability to communicate technical concepts to both technical ...

strong plus Technical Skills: • Proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL) • Familiarity with data pipeline and workflow management tools (e.g., Apache Airflow) • Experience with programming languages such as Python, Java, or Scala. Python is highly preferred • Basic understanding of cloud platforms and services (e.g. ...

Snowflake Production experience with dbt (mandatory) Strong SQL and Python programming skills Experience with Git-based workflows and DevOps practices Familiarity with orchestration tools (Airflow, Prefect) and ETL/ELT patterns Knowledge of cloud platforms (AWS, Azure) and security best practices ...

Snowflake Production experience with dbt (mandatory) Strong SQL and Python programming skills Experience with Git-based workflows and DevOps practices Familiarity with orchestration tools (Airflow, Prefect) and ETL/ELT patterns Knowledge of cloud platforms (AWS, Azure) and security best practices ...

modular, reusable transformations using SQL and Python. Implement CI/CD pipelines and manage deployments via Git. Automate workflows using orchestration tools such as Airflow or dbt Cloud. Configure and optimise Snowflake warehouses for performance and cost efficiency. Required Skills & Experience 7+ years in data engineering roles. 3+ years ...

modular, reusable transformations using SQL and Python. Implement CI/CD pipelines and manage deployments via Git. Automate workflows using orchestration tools such as Airflow or dbt Cloud. Configure and optimise Snowflake warehouses for performance and cost efficiency. Required Skills & Experience 7+ years in data engineering roles. 3+ years ...

Escalate complex requests to specialized engineering pods and track to closure. TECHNICAL IMPLEMENTATION & ENABLEMENT Provision and configure research, data, and trading environments (Python, AWS, Airflow, GitHub, Coder, etc.). Coordinate bespoke data ingestion (market, alternative, proprietary) via APIs, direct feeds, and file transfers. Deliver live, role-based training ...

technologies. Experience with interface/API data modelling. Experience with CI/CD (GitHub Actions or similar). AWS data engineering certifications. Knowledge of Airflow, DBT, Iceberg tables and metadata/catalogue tools such as Atlan. What’s on Offer A comprehensive benefits package supporting physical, emotional and financial ...

deliver high-quality, reliable data. Governance & Quality: Use Dataplex for cataloguing, classification, and governance; enforce data standards and validation. Automation & Orchestration: Design workflows in Apache Airflow and automate data quality checks, schema changes, and lifecycle processes. Continuous Improvement: Champion best practices, mentor junior engineers, and push for scalable … Dataproc, Dataplex. Strong development skills in Java and Python. Experience with weblog ingestion & processing, both real-time and batch. Solid hands-on use of Apache Airflow. Familiarity with AWS Kinesis Firehose for log/data ingestion. Knowledge of data governance, metadata management, and stewardship. Comfort working with clickstream/ ...

maintain highly reliable and well-tested data and ML pipelines to extract, transform, and structure large datasets for ML applications. Expertise in using Apache Airflow (or similar orchestration tools like Prefect/Dagster) to define, schedule, and monitor complex data and ML workflows (DAGs). Implement comprehensive software … engineering teams. Bonus Skills to Stand Out (Optional): Experience with CI/CD tooling (e.g., GitHub Actions, Jenkins) specifically for ML pipelines and Airflow DAG deployment. Experience with data quality monitoring tools and frameworks. Master's or PhD in Computer Science, Data Science, or a related field. Relevant certifications ...

governance, and data architecture. Familiarity with modern data architectures and cloud-based/distributed systems. Proficiency in SQL, Python and data pipeline tools (e.g., Apache Airflow, Spark). Experience working with major cloud platforms (AWS, Azure, GCP) and big data technologies. Awareness of data governance, data sharing across ...

Design, SQL and database optimizations Experience of being a bar-raiser and helping improve engineering best practices Experience of orchestration tools like Databricks or Airflow Experience with cloud data warehouses like Snowflake or BigQuery Coaching, mentoring and growing an established mid-level team who are eager to learn from ...

performance, cost efficiency, and scalability.* Troubleshoot and resolve data pipeline issues swiftly and effectively across the data platform.* Work with orchestration tools such as Airflow, ADF, or Prefect to schedule and automate workflows.* Keep abreast of industry trends and emerging technologies in data engineering, and continuously improve your skills ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...

strong ability to learn it quickly. Strong SQL skills; working familiarity with Python for scripting and DBT automation. Experience integrating DBT with orchestration tools (Airflow, Dagster, Prefect, etc.). Solid understanding of modern data engineering, ELT patterns, and version-controlled analytics development. Preferred Prior experience operationalizing ML workflows inside ...