1 to 25 of 32 Apache Airflow Jobs in London

applications, working closely with data engineers, analysts, and platform teams. Day to day duties Developing and maintaining data pipelines and processing jobs using Apache Spark and Python Orchestrating workflows with Apache Airflow Provisioning and managing cloud infrastructure on AWS using Terraform (Infrastructure as Code) Containerising applications … teams in an Agile environment, contributing to design reviews, code reviews, and documentation Essential Skills & Experience Strong hands-on experience with Python Experience with Apache Spark for large-scale data processing Practical knowledge of AWS services (eg S3, Lambda, EC2, Glue, RDS - adjust as relevant) Experience writing and maintaining ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

working with Azure or AWS cloud platforms Experience with modern data platforms such as Databricks, Snowflake or Redshift Experience using orchestration tools such as Airflow, Azure Data Factory or Prefect Strong understanding of data warehousing and data modelling principles Excellent problem-solving skills with a delivery-focused mindset Comfortable ...

Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools such as Airflow, ADF, or equivalent. Understanding of data governance, data quality, and security practices. Familiarity with CI/CD pipelines and version control tools. Preferred Skills ...

processes and data pipeline development Experience working with cloud platforms such as AWS, Azure or GCP Knowledge of technologies such as Spark, Kafka, Airflow or similar Experience with databases, data modelling and data warehouse solutions Understanding of CI/CD, software engineering best practices and secure development Strong communication ...

maintaining scalable data pipelines and integrations. Developing robust ETL and data transformation processes to support analytics and reporting requirements. Working with Snowflake, DBT, Apache Airflow, AWS, SQL and Python to deliver high-quality data solutions. Collaborating with Data Architecture, DevOps and business stakeholders to ensure trusted and accessible … ideally within AWS. Data modelling, transformation and orchestration methodologies. Delivering high-quality, well-documented and thoroughly tested data solutions. Experience with Snowflake, DBT and Apache Airflow. Interest in AI and emerging technologies within the data engineering landscape. THE BENEFITS: You will receive a salary ...

Optimise Python performance through effective use of loops , vectorization , threading , and understanding of the Global Interpreter Lock (GIL) . Desirable Experience Experience with Kubeflow , Airflow , or dbt . Hands-on experience with Google Cloud Platform (GCP) . Knowledge of containerisation (Docker, Kubernetes) and CI/CD practices. Familiarity with ...

marketing teams to align data solutions with business needs Requirements Strong SQL skills and experience with ETL/ELT and pipeline orchestration tools (e.g., Airflow, dbt, Data Factory) Experience with cloud data platforms (e.g., Azure, AWS, or GCP) Proficiency in data modelling (dimensional, entity-relationship, event-driven) Understanding ...

identify anomalies and ensure data integrity for use by commercial trading and analytics teams. Automate data workflows using Python, SQL, and orchestration tools (e.g., Airflow or similar). Assist in transitioning legacy datasets and codebases into scalable, cloud-native workflows aligned with our modern data architecture. Document data sources ...

initial 6-month contract. Core Technical Requirements Strong experience with Snowflake for data warehousing, including writing efficient SQL and managing schemas. Proficiency in Airflow for orchestration and workflow management. Hands-on experience with AWS services , particularly S3 for storage and Lambda for serverless processing. Familiarity with Kafka concepts (producers ...

communication skills. Highly Desirable: Experience with Spark/PySpark. Familiarity with Azure Data Lake, Snowflake, or similar platforms. Knowledge of orchestration tools like Airflow or Azure Data Factory. Exposure to financial services environments. Experience supporting large-scale data transformation or migration programmes. What We're Looking For: A senior ...

data platforms, migrating legacy environments into cloud-native solutions using the latest AWS and Snowflake technologies. Key skills: Snowflake, Teradata & Informatica Python & SQL AWS, Airflow & DBT GitHub & CI/CD ETL & Data Warehousing Control-M, Unix/Linux Data Modelling & Performance Optimisation You'll design and optimise scalable data ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers and semantic models Data Migration & Mapping Demonstrated ...

tools. Experience building and optimising scalable data pipelines and data warehouses. Knowledge of AWS technologies such as Redshift, S3 and Athena. Experience with Airflow, Fivetran or similar orchestration and integration tools. Strong Linux/Unix and Bash scripting skills. A solid understanding of data quality, governance, lineage and monitoring. ...

modern cloud based data platforms. Solid understanding of data modelling and data transformation principles. Ability to work effectively within collaborative engineering teams. Desirable Experience: Airflow Databricks Experience supporting AI or machine learning initiatives. Semantic layer experience. Exposure to Azure and multi cloud environments. Understanding of Inmon 3NF data modelling ...

join their London-based team. Key Responsibilities: Design and maintain scalable data pipelines across diverse sources. Automate and optimise workflows using tools like Airflow, dbt, and Spark. Support data modelling for analytics, dashboards, and A/B testing. Collaborate with cross-functional teams to deliver data-driven insights. Work ...

improvements What will set you apart Hands-on experience with the modern data stack, for example Iceberg or Delta, Snowflake or Databricks, dbt or Airflow Experience with a data migration or re-platforming project, on either the product or the engineering side Experience in financial services or another regulated ...

learning services. Experience with machine learning frameworks such as TensorFlow, PyTorch, or scikit-learn. Knowledge of pipeline orchestration and data workflow tools such as Airflow, Dataform, or dbt. Experience delivering advanced measurement and modelling solutions, including marketing mix modelling, incrementality testing, causal impact analysis, attribution modelling, propensity modelling, churn ...

organisation Deep understanding & hands-on leverage of Modern, Cloud scale Data Technologies, with specific expertise in Distributed Data Processing architectures (Spark, PySpark, Python, Airflow), ELT data processing underpinning Data LakeHouses and Data Products for scaled use within ML & AI Expertise in Streaming Data, Event Driven Design, Data Governance ...

Backend Engineering background Experience building APIs and system integrations Experience working within complex, data-intensive environments Strong understanding of software engineering best practices Desirable Airflow Snowflake AWS Exposure to data warehousing and modern analytics platforms Experience supporting reporting, commercial analytics, or performance data solutions IDEAL BACKGROUND We're especially ...

background as a Machine Learning Engineer. Technical Skills: Strong in SQL and Python (Pandas, Scikit-learn, Jupyter, Matplotlib). Data transformation & manipulation : experience with Airflow, DBT and Kubeflow Cloud: Experience with GCP and Vertex AI (developing ML services). Expertise: Solid understanding of computer science fundamentals and time-series ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs. Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB … lakehouse architectures.- Familiarity with big data processing frameworks like Spark. Data Engineering Knowledge:- Proven experience building ETL/ELT pipelines and using orchestration tools (Airflow, Azure Data Factory).- Understanding of JSON, XML, APIs, and data ingestion technologies. Cloud Experience:- Familiarity with cloud platforms (Azure ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB … lakehouse architectures. - Familiarity with big data processing frameworks like Spark. Data Engineering Knowledge: - Proven experience building ETL/ELT pipelines and using orchestration tools (Airflow, Azure Data Factory). - Understanding of JSON, XML, APIs, and data ingestion technologies. Cloud Experience: - Familiarity with cloud platforms (Azure ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...