51 to 75 of 195 Apache Airflow Jobs in the UK

infrastructure, and orchestration. What You’ll Be Doing Develop and maintain scalable data solutions using Snowflake Design and automate ETL/ELT pipelines with Airflow or Dagster Implement and manage infrastructure as code using Terraform Drive performance tuning, cost optimisation, and platform scalability Collaborate closely with data scientists, engineers ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

communication skills. Highly Desirable: Experience with Spark/PySpark. Familiarity with Azure Data Lake, Snowflake, or similar platforms. Knowledge of orchestration tools like Airflow or Azure Data Factory. Exposure to financial services environments. Experience supporting large-scale data transformation or migration programmes. What We're Looking For: A senior ...

systems Excellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams. Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructure Experience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones. What ...

software engineering skills in Python, with experience building scalable services, APIs, and production-quality code Experience with modern ML tooling and platforms (e.g.Databricks,MLflow, Airflow, Kubeflow, SageMaker, Vertex AI) Familiarity with CI/CD pipelines and infrastructure-as-code (e.g.Terraform, CloudFormation) Experience building robust, maintainable, and testable ML pipelines ...

deep knowledge in core processing and orchestration products such as Big Query, Data Flow, Data Fusion, Data Stream, Cloud Functions, Data Proc and Airflow/Composer. You will have excellent problem-solving skills, a rigorous approach to code checks/peer reviews and have the strength of character ...

technologies and practices such as Docker, Terraform, CI/CD pipelines and infrastructure‐as‐code. Familiarity with MLOps tools and practices such as MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark ...

governance, traceability, and audit requirements considered from the outset. Hands-on experience building and owning production data pipelines, with strong familiarity with dbt and Airflow or equivalent orchestration tools. Full mastery of SQL including complex joins, CTEs and analytical functions, alongside strong Python proficiency for pipeline development and analysis. ...

implementation of modern data architectures and concepts such as cloud services (e.g., AWS, Azure, GCP), real-time data distribution (e.g., Kafka, Kinesis, DataFlow, Airflow), NoSQL (e.g., MongoDB, DynamoDB, HBase, CosmosDB), GraphQL and modern data warehouse tools including Snowflake and DataBricks* Ability to think strategically and relate architectural decisions ...

Excellent verbal and written communication skills BS or MS degree in Computer Science or equivalent Experience in distributed computing frameworks like Hive/Hadoop, Apache Spark Experience in developing Finance or HR related applications Experience with cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic Cache … Lambda Working experience with Terraform Experience in creating workflows for Apache Airflow What the job involves As a senior technical lead, you consider yourself a platform services developer and develop innovative solutions using modern software frameworks and paradigms You are self‐driven and a problem solver, with ...

query optimisation. Redis - caching, pub/sub, distributed locks and idempotency patterns. Serverless Framework v3, Docker (container-based Lambdas), Terraform, Jenkins CI/CD. Apache Airflow DAG authoring and AWS Glue with PySpark for batch ETL. pytest and moto (offline AWS service mocking), Black formatter, ruff linter. Datadog … retry patterns. Hands-on AWS infrastructure experience: Serverless Framework v3, Terraform, ECR, IAM. Experience deploying Docker container-based Lambda functions. Data engineering background with Airflow DAGs and PySpark/Glue ETL pipelines is an advantage. Multi-tenant SaaS platform experience at scale is desirable. Familiarity with telecom, OEM device ...

Data Engineer to design, build, and maintain scalable data pipelines powering Moody’s next-generation digital content platform. You will work with DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS, driving platform optimizations and cross-functional collaboration. In this role, you’ll collaborate with product owners ...

Strong cloud data platform expertise, including Google Cloud Platform (GCP). Hands-on experience with BigQuery, Dataflow, Datastream, Data Fusion, Cloud Functions, Dataproc, and Airflow/Composer. Advanced SQL and Python skills. Expertise in data lakes, data warehouses, and modern data architectures. Strong understanding of data governance, lineage ...

tools. Experience building and optimising scalable data pipelines and data warehouses. Knowledge of AWS technologies such as Redshift, S3 and Athena. Experience with Airflow, Fivetran or similar orchestration and integration tools. Strong Linux/Unix and Bash scripting skills. A solid understanding of data quality, governance, lineage and monitoring. ...

database querying languages Experience in large‐scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Proficiency in automation, continuous delivery methods, and all aspects of the Software Development Life Cycle Advanced understanding of agile methodologies, CI/CD, application ...

production-grade data pipelines Solid understanding of integration patterns including REST APIs, webhooks, OAuth and event-driven architectures Experience with orchestration tools such as Airflow, Prefect or Dagster Familiarity with cloud environments (AWS, GCP or Azure) Experience with containerisation tools such as Docker and Kubernetes Proven experience integrating multiple ...

Practical experience with Azure (Azure OpenAI, Azure AI Foundry, AI Search) and familiarity with AWS or GCP Proficiency with containerized deployment, MLOps tooling (MLFlow, Airflow), and infrastructure‐as‐code practices Bachelor’s or Master’s degree in Computer Science, Data Science, or related field Nice to have Experience with ...

database querying languages Experience in large‐scale data processing, microservices, API design, Kafka, Redis, MemCached, Observability tools (Dynatrace, Splunk, Grafana), and Orchestration tools (Airflow, Temporal) Proficiency in automation, continuous delivery methods, and all aspects of the Software Development Life Cycle Advanced understanding of agile methodologies, CI/CD, application ...

interpersonal skills Preferred qualifications: Experience in the gaming or entertainment industry Experience working with tools and technologies such as Spark, Snowflake, Qwak, MLFlow, Airflow, etc Contributions to open‐source AI/ML projects or participation in AI/ML competitions (e.g., Kaggle) Experience with training and fine‐tuning large ...

interpersonal skills Preferred qualifications: Experience in the gaming or entertainment industry Experience working with tools and technologies such as Spark, Snowflake, Qwak, MLFlow, Airflow, etc Contributions to open‐source AI/ML projects or participation in AI/ML competitions (e.g., Kaggle) Experience with training and fine‐tuning large ...

understanding of cloud infrastructure, observability, monitoring, incident response, and operational best practices relevant technologies (we use): AWS incl. S3, ECS, API Gateway, Batch, Redshift, Airflow, GitHub Actions, Docker, Pulumi, Python, Javascript/react high ownership mentality who can work across the stack with a primary responsibility for DevOps reliability ...

reproducibility, dataset versioning, and efficient access Working knowledge of cloud platforms (AWS, GCP, or Azure) and modern MLOps tooling (e.g. MLflow, Weights & Biases, DVC, Airflow) Strong experience with Python and associated scientific software packages such as PyTorch, OpenCV, Pandas, SciPy, NumPy, SciKit-learn, etc. Strong software engineering practices including ...

Styled Components), React, Redux, Redux Hooks, and other modern state management libraries. Kotlin on Android and Swift on iOS. Data: SQL, Python, Snowflake, dbt, Airflow & Looker AI at Marshmallow: Sierra.ai for conversational AI, Python and LangChain for agentic AI solutions, Sagemaker and MLflow for Machine Learning, Cursor and Claude ...

requirements into Data Science deliverables. The opportunity to work with leading data engineering technologies, including Google Vertex AI tools, Google BigQuery, BigQuery Pipelines, and Airflow, with leading Data Science tools and emerging technologies for model building and deployment. Opportunities to develop your career in a friendly, diverse, innovative, international ...

engineering mindset and experience building production-grade systems Experience developing ETL processes and large-scale financial data pipelines Strong Azure experience Docker and Kubernetes Apache Airflow GitHub and CI/CD (GitHub Actions) Experience building cloud-native research or production platforms Experience working with market data, security master ...