276 to 300 of 315 Apache Airflow Jobs in London

risk through data, building robust pipelines and driving insights to support Wise's secure and compliant framework. You will work with Python, DBT, Airflow and REST APIs to create scalable data solutions, collaborate across security domains, and define KPIs for continuous monitoring and improved efficiency. #J-18808-Ljbffr ...

building internal developer tools, DSLs, or compilers, multi-language proficiency such as Go, C, or Rust, cybersecurity, workflow orchestration with Prefect or Airflow, high-growth SaaS Условия: Competitive salary Meaningful equity One of the first engineering hires Direct access to the founders Huge ownership from day one Build products ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools … partners across the organization via Technical Design Documents. This is a great opportunity to further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform & Google Cloud Platform. Benefits Hybrid working Long contract Award winning work environment ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools … partners across the organization via Technical Design Documents. This is a great opportunity to further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform & Google Cloud Platform. Benefits Hybrid working Long contract Award winning work environment ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code” tools … partners across the organization via Technical Design Documents. This is a great opportunity to further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform & Google Cloud Platform. Benefits Hybrid working Long contract Award winning work environment ...

/Java and/or Python. Experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of SQL and shell scripting; Experience with Docker/k8s and "infrastructure as a code" tools … partners across the organization via Technical Design Documents. This is a great opportunity to further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform & Google Cloud Platform. Benefits Hybrid working Long contract Award winning work environment #J-18808-Ljbffr ...

global market. About the role We are looking for a Lead Data Engineer to take ownership of a business domain within our modern Snowflake, Airflow, and dbt data platform. This role will combine hands-on technical leadership with team coordination, helping ensure stable platform operations, strong data quality … leading small data engineering teams, ideally 2–3 engineers. Experience designing, building, and maintaining data processing systems at scale. Experience with Azure cloud services, Airflow, dbt, Kubernetes, GitHub, and Terraform. Experience working in fast-moving, high-growth, or startup-like environments is preferred. Strong communication skills with the ability ...

integration failures, data inconsistencies, and production issues Key Skills Python & SQL (production-grade pipeline development) REST APIs, webhooks, OAuth, event-driven architecture Orchestration tools: Airflow, Prefect, or Dagster Cloud platforms: AWS, GCP, or Azure Docker & Kubernetes Microsoft 365 & Microsoft Copilot Desirable Skills Vector databases and embedding pipelines Real-time … robust, production-grade data pipelines from scratch. Deep familiarity with integration patterns: REST APIs, webhooks, OAuth, and event-driven architectures. Experience with orchestration tools (Airflow, Prefect, or Dagster) and transformation frameworks (dbt or similar). Comfortable across cloud platforms (AWS, GCP, or Azure) and with containerisation (Docker, Kubernetes). ...

Develop batch and ELT/ETL data workflows for ingesting and processing large datasets* Work with technologies such as Python, Java, Spark, SQL, and Airflow* Apply good software engineering practices, including testing, documentation, and CI/CD* Contribute to cloud‐based systems (AWS), focusing on reliability, monitoring, and automation … data engineering concepts such as data lakes, batch processing, and distributed systems* Experience building or operating data pipelines using tools like Spark and Airflow* Familiarity with cloud platforms (AWS preferred) and production monitoring* Clear communication skills and a collaborative approach to problem‐solving.**Why join us?**Join our team ...

order to build scalable and robust solutions Utilize Docker to create containerized microservices Automate infrastructure configuration using Terraform and Puppet Use tools such as Airflow and Snowflake to build data pipelines Integrate various JavaScript libraries such as React to deliver seamless user experiences Design data models and persist data … Server Use T‐SQL to manipulate and transform data Utilize Kafka and SQS for messaging Improve search using Apache Solr Build caching layers using Redis Use TeamCity and Jenkins to automate deployments Write automated unit, integration and deployment tests Build tooling to automate away repetitive tasks Qualifications Blackstone seeks ...

Docker An interest or experience in data/network security would be a nice-to-have Datadog/Grafana/Prometheus Data related products (Airflow, Jupyter, Spark, dbt etc) The projects will be varied and we're looking for someone who can work autonomously and proactively to scope problems … Stack... Python as our main programming language Terraform for our infrastructure definition Kubernetes for data services and task orchestration Argo CD for application deployments Airflow purely for job scheduling and tracking CircleCI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Postgres/Aurora ...

days on site A London Trading firm is building a trading data platform from scratch, in house, on open source. Spark, Kafka, Polaris, Ranger, Airflow, MinIO. This is the person who builds it. Why now The analytics function is running on legacy warehouses that were never built … ingestion pipelines for both streaming and batch sources Write the Spark jobs that clean, transform and land data in the lakehouse Build and own Airflow orchestration Work on cataloguing and access control with Polaris and Ranger Help migrate data off the legacy warehouses and retire them What we need ...

long-term fixes TypeScript and Node.js sit at the heart of the wider application stack, including the services and APIs surrounding the data platform. Airflow orchestrates pipelines and workflows, with Python also used across data processing and automation. Everything runs within a modern Azure environment. You're are encouraged … APIs around the resulting data Are confident with TypeScript and/or Python Understand relational, NoSQL and modern cloud-based storage Have worked with Airflow or similar orchestration tooling Care about data quality, clean engineering and genuine ownership Have experience building SaaS products or other production platforms Experience with ...

engineering & the platform Build and maintain ingestion and transformation pipelines across our stack. We use a modern data stack, such as AWS, Redshift, and Airflow Write production-quality Python for extraction, transformation, and automation tasks that fall outside dbt’s remit Help us modernise: we’re actively evolving this … plain language Ownership: you’re comfortable being the first and only person in this role, and building things properly from scratch Nice-to-haves Airflow or similar orchestration tools, and familiarity with data ingestion tools such as dlt or Airbyte Warehouse cost and performance tuning, particularly on Redshift Fintech ...

We tackle the most complex problems in quantitative finance, by bringing scientific clarity to financial complexity. From our London HQ, we unite world-class researchers and engineers in an environment that values deep exploration and ...

data scientists. How to make an Impact Architect and Implement : Own the technical roadmap for our Spark-based data processes to ensure our Airflow pipelines are performant, cost-effective, and scalable. You will enhance our existing services through hands-on development and solve any complex performance bottlenecks, concurrency issues … deployment. Expert-level Python and its data ecosystem (Numpy, Pandas), including designing frameworks for data tasks. Deep understanding of distributed data processing engines like Apache Spark. Ability to reason about the full data lifecycle from ingestion through to real-time serving - your system designs will need to meet downstream ...

What will you be working on? Data Engineering Build and Optimise Pipelines: Develop, optimise, and maintain data pipelines using Python, REST APIs, DBT, and Airflow to ensure seamless data flow for compliance and risk management. Understanding Control Requirements & Continuous Monitoring: Partner closely with control owners across security domains (such … organisation. QualificationsTechnical Expectations: Proficient in Python, with experience in automation and building data pipelines. Strong expertise in working with tools like DBT and Airflow for ETL processes. Skilled in using REST APIs to develop and manage data integrations. Proficient in SQL and familiar with data visualization tools such ...

Data Engineering team, which builds and governs the data foundation that powers the business. You'll work within our stack — Python ingestion pipelines, Airflow orchestration, and Snowflake/Databricks — helping move data reliably and securely from source to decision-ready output.You'll implement features and fixes against a given … with solid experience building and operating production data pipelines Comfort investigating and root-causing pipeline issues independently before escalating Experience with workflow orchestration tools (Airflow or similar) and a cloud data warehouse/lakehouse (Snowflake, Databricks, or similar) Solid understanding of data pipeline concepts: idempotency, schema evolution, backfills ...

requiring strong data engineering expertise, experience working with financial data, and the ability to operate within a fast-paced investment environment. Key Skills Snowflake Airflow DBT Bloomberg Integrations Responsibilities Design, build and maintain scalable data pipelines and workflows. Develop and enhance data solutions using Snowflake, Airflow and DBT. … Drive best practices across data engineering, automation and platform support. Required Experience Experience within Investment Management, Asset Management or Financial Services Expertise with Snowflake, Airflow and DBT. Experience integrating Bloomberg data and services into enterprise platforms. Strong SQL and data modelling skills. Experience working with large-scale financial ...

We are Foundational. Foundational is an early-stage, mission-driven startup solving one of the hardest engineering challenges of the decade: creating an exquisite, real-time understanding of Earth and orbit using ground-based lasers. ...

Why Sony Interactive Entertainment? Sony Interactive Entertainment isn't just the Best Place to Play — it's also the Best Place to Work. Sony Interactive Entertainment (SIE) is the company behind the PlayStation brand. As ...

driven user experience. Hybrid role with three on-site days in City of London. Responsibilities include building scalable data pipelines (BigQuery, Dataflow/Apache Beam, Airflow), ensuring data quality and observability (Looker, Monte Carlo), and collaborating with product engineering and data science teams to plan data tracking ...

value, and/or FICC options trading Experience with Linux-based, concurrent, high-throughput, low-latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelor or advanced ...

value, and/or FICC options trading Experience with Linux‐based, concurrent, high‐throughput, low‐latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced ...

store of value, and/or FICC options tradingExperience with Linux-based, concurrent, high-throughput, low-latency software systemsExperience with pipeline orchestration frameworks (e.g. Airflow, Dagster)Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databasesHave a Bachelors or advanced degree ...