10 of 10 Apache Airflow Jobs in the City of London

create the queries and processes necessary to move everything to the new. As migration work wraps up, the role will transition to leveraging Airflow and other tooling to automate operational and financial processes. This is a great role for exceptionally outgoing individuals as all work will require extensive collaboration … design the data process' to be run the day each partner is migrated Build and maintain ETLs and workflow automation using Snowflake and Apache Airflow Thoroughly document all learnings and work-in-progress using version control Implement data quality checks and reconciliations at all steps in the process ...

driving innovation in data engineering. Desirable Criteria Experience delivering data pipelines on Hortonworks/Cloudera on-prem and leading cloud migration initiatives. Familiarity with Apache Airflow, data modelling, and metadata management Experience influencing enterprise data strategy and contributing to architectural governance. To apply for this position please submit ...

familiarity with data preparation, transformation, visualisation, and cloud deployment. Experience with Google Cloud Platform and tools such as Google Pub/Sub, Dataflow, Dataproc, Airflow, Dataprep, Data Fusion. Experience with CI/CD and DevOps tooling (e.g., Jenkins, Kubernetes). Experience working within Agile delivery models (Scrum, Kanban), including ...

discipline Solid Python programming skills Good working knowledge of SQL Comfortable using Git for version control Desirables: Exposure to workflow orchestration tools (e.g. Prefect, Airflow, Dagster) Experience with cloud data warehouses (Azure SQL, Snowflake) or dbt Basic familiarity with Docker and BI tools (Power BI, Tableau) Interest in shipping ...

Python, and Ansible Familiarity with containers (Docker, Kubernetes, LXC) Experience with monitoring tools and SQL databases Exposure to big data systems (e.g., Spark, HDFS, Airflow) Summary A great opportunity for someone who enjoys solving complex infrastructure challenges, automating systems, and working closely with engineering teams. Spectrum IT Recruitment (South ...

production environments; Python expertise - idiomatic, well-tested, production-grade code, not notebook scripts * ETL/ELT pipeline design and implementation at scale; orchestration with Airflow, Prefect, or equivalent; reliability-first mindset including backfill, retry, and exactly-once semantics * Azure data platform - Azure Data Factory, Azure Databricks, Azure Synapse Analytics … documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building and maintaining ...

implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality controls Design and manage secure data ingestion frameworks (e.g. Apache NiFi, APIs, SFTP/FTPS) for internal and external data sources Architect and maintain secure AWS-based data infrastructure (S3, IAM, KMS, Glue, Lake … Formation, Lambda, Step Functions, CloudWatch, etc.) Implement orchestration using tools such as Airflow, Databricks Workflows, and Step Functions Champion data quality, observability, and reliability (SLAs, monitoring, alerting, reconciliation) Drive CI/CD best practices for data platforms (infrastructure as code, automated testing, versioning, environment promotion) Mentor engineers on distributed ...

moving away from legacy systems towards a scalable, cloud-native stack. What you'll be working on: Building and maintaining scalable data pipelines using Airflow and dbt Working across AWS cloud infrastructure Collaborating with analysts and data scientists to deliver reliable, clean data for downstream reporting and business decision … governance, and best practices across the team What they're looking for: Proven data engineering experience with strong Python and SQL Hands-on with Airflow and/or dbt Solid AWS experience Experience with ETL pipeline development at scale Comfortable working in Agile/Scrum environments Full right ...

root cause analysis, managing SLAs, and improving resiliency across business-critical data feeds. Requirements: SQL Python/Bash ETL/ELT support Workflow orchestration (Airflow or Dagster) AWS data services (Athena, S3, MWAA, Iceberg) Data quality monitoring & incident management Nice to Have: Airbyte DBT Apache Spark Grafana ...

security. Drive modernisation by transitioning from legacy systems to a lean, scalable platform. Act as a lead expert for technologies such as AWS, DBT, Airflow, and Databricks. Establish best practices for data modelling, ingestion, storage, streaming, and APIs. Governance & Standards Ensure all technical decisions are well-justified, documented … engineering, including data ingestion, transformation, and storage. Significant hands-on experience with AWS and its data services. Expert-level skills in SQL, Python, DBT, Airflow and Redshift. Confidence in coding, scripting, configuring, versioning, debugging, testing, and deploying. Ability to guide and mentor others in technical best practices. A product ...