76 to 100 of 281 Apache Airflow Jobs in London

analyse large datasets and communicate insights effectively. Experience with LLMs, generative AI, and prompt engineering. Knowledge of MLOps tools and frameworks (MLflow, Kubeflow, Airflow). Familiarity with SQL, NoSQL, and distributed data technologies. Experience deploying AI models in production environments. Publications, patents, or open-source contributions in ML/ ...

another language used for data processing Batch and event-driven processing. Experience with Kafka or an equivalent streaming system, a workflow orchestrator such as Airflow, Dagster or Prefect, and transformation tooling such as dbt or equivalent SQL-based workflows Cloud warehouse, lake or lakehouse experience , and integrating data from ...

implementation of modern data architectures and concepts such as cloud services (e.g., AWS, Azure, GCP), real-time data distribution (e.g., Kafka, Kinesis, DataFlow, Airflow), NoSQL (e.g., MongoDB, DynamoDB, HBase, CosmosDB), GraphQL and modern data warehouse tools including Snowflake and DataBricks Ability to think strategically and relate architectural decisions ...

closely related Financial Services organisation, with an understanding of the data, operational and regulatory challenges that underpin investment platforms. Proven hands on experience owning Airflow and dbt end-to-end, including designing DAGs and dbt projects, setting standards, and running, upgrading and supporting them end to end. Proven experience ...

operating ETL/ELT pipelines and large-scale data architectures. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). Ability ...

skills; experience with Oracle databases . Experience designing and supporting ETL/ELT pipelines and data quality frameworks . Working knowledge of Autosys& Apache Airflow Experience with CI/CD tools (Git, Harness, UrbanCode Deploy (UCD), Red Hat OpenShift) Familiarity with AWS S3 for large-scale data storage. ...

production problems and taking responsibility for critical infrastructure. Startup Mindset: Proactive, resourceful and comfortable taking ownership in a fast-moving engineering environment. Bonus : with Apache Airflow, real-time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial-market systems. Benefits An Ambitious, Collaborative Culture: Join ...

self-motivated, inquisitive mindset and a methodical, agile approach to problem-solving Solid Linux administration and networking skills, plus Git-based workflows Familiarity with Apache Airflow, Dask and JupyterHub Experience with modern build systems (especially Bazel) and CI/CD pipelines. What we offer At Macquarie ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

database querying languages Experience in large-scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Proficiency in automation, continuous delivery methods, and all aspects of the Software Development Life Cycle Advanced understanding of agile methodologies, CI/CD, application ...

systems, orchestration, governance, and scalable data product design Experience with technologies and ecosystems such as GCP, AWS, Azure, Databricks, Snowflake, Kafka, Spark, dbt, Kubernetes, Airflow, ML/AI enablement, or similar modern platform tooling Strong commercial acumen and ability to engage credibly with senior technical and business stakeholders Experience ...

/ELT frameworks, and cloud data platforms such as Azure, AWS, or GCP. Data modelling and pipeline orchestration using tools such as Airflow, Data Factory, dbt, or similar. Integrate CDP data sources across digital, CRM, transactional, and behavioural data; support identity resolution, consent handling, audience activation, journey events, trigger ...

structured, methodical approach.* Excellent communication and collaboration abilities.**Preferred Experience:*** Building or migrating to **Snowflake** (or other cloud-based) data platforms.* Familiarity with **Dagster**, **Airflow**, or similar orchestration tools.* Supporting SQL Server solutions, including administration* Experience managing the transition between legacy and modern data systems.* **Exposure to financial data ...

partnering with data science/analytics teams. Relevant certifications (PMP, SAFe, Scrum, AWS/Azure) or experience with core stack (Databricks/Snowflake, Kafka, Airflow, AWS/Azure) are a plus. #J-18808-Ljbffr ...

Snowflake and dbt experience: modelling, semantic layers, agents & skills, testing, performance and cost tuning. Production experience with a pipeline orchestrator, ideally Dagster (or similar Airflow, Prefect), including dependent workloads, partial failures, and observability. Modern cloud infrastructure (Azure/AWS/GCP) and Kubernetes experience deploying and operating containerised workloads ...

production (not just training models in notebooks), including familiarity with serving patterns (e.g., REST APIs, batch inference, event-driven) Experience with orchestration tools (e.g., Airflow, Dagster) and cloud-native ML platforms (e.g., AWS SageMaker, GCP Vertex AI) Ability to influence decision-making regarding infrastructure and tooling, balancing 'build vs. ...

various platforms Experience briefing, designing and implementing ETL solutions end-to-end, from requirements gathering through to deployment Experience with data orchestration tools (e.g. Airflow, Prefect) and building scalable data pipelines Strong Git and collaborative development skills (e.g. GitHub, GitLab) Strong understanding of data fundamentals: MDM, data integrity best ...

Spark and Databricks AWS services (e.g. IAM, S3, Redis, ECS) CI/CD tooling and best practices Streaming and batch data systems (e.g. Kafka, Airflow, RabbitMQ) Additional Information Health + Mental Wellbeing PMI and cash plan healthcare access with Bupa Subsidised counselling and coaching with Self Space Cycle ...

assistants as part of the software development lifecycle.* Experience designing, building, and operating data pipelines and ETL/ELT workflows using Databricks, dbt, and Apache Airflow.**Preferred Skills & Experience*** Experience with frameworks like ReactJS, Angular, or Vue.js to build interactive and responsive user interface.* Experience building or designing scalable ...

Python, Node.js, Java), frontend (React, Angular), APIs (REST/GraphQL) Machine Learning: Foundations of MLflow, Weights & Biases, LoRA, PEFT, SageMaker, Vertex AI, AzureML, Kubeflow, Airflow, Prefect, TorchServe, Triton, vLLM, SHAP, LIME, pandas, Spark, SQL LLM tooling: Familiarity with LangChain, Semantic Kernel, prompt chaining, vector search, and context management Performance ...

experience as a Data Engineer or Senior Data Engineer Expert-level Python development skills Extensive hands-on experience with Databricks in enterprise environments Strong Apache Spark and PySpark expertise Advanced SQL development and optimisation skills Experience designing and building large-scale ETL and data integration solutions Strong understanding … environments Exposure to risk, trading, treasury or regulatory data domains Experience with Kafka, streaming technologies or real-time data processing Databricks certifications Experience with Airflow, Databricks Workflows or similar orchestration tools CI/CD pipeline development and DevOps practices Docker and Kubernetes Experience supporting data science or machine learning ...

environments. Technical Skills: Experience building and operating ETL/ELT pipelines. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

operating ETL/ELT pipelines and large-scale data architectures. Deep hands‐on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with similar technologies is essential). ...

Familiarity with PostgreSQL and Snowflake Preferred: Familiarity with Web Frameworks such as Django, Flask or FastAPI Preferred: Familiarity with event streaming platforms such as Apache Kafka Preferred: Familiarity with data pipeline platforms such as Apache Airflow Preferred: Familiarity with Java Preferred: Experience in one or more relevant ...

tooling, automated testing and reliable deployment processes. Startup Mindset: Proactive, resourceful and comfortable taking responsibility in a small, fast‐moving engineering team. Experience with Apache Airflow, Kubernetes, real‐time data platforms, AI/ML infrastructure, LLM agents, GPU infrastructure or financial‐market systems would be useful ...