26 to 50 of 502 Apache Airflow Jobs in the UK

Amazon Redshift AWS Lambda Amazon Athena Amazon RDS AWS IAM Amazon CloudWatch Experience with services such as Kinesis, EMR, Step Functions or MWAA/Airflow would be advantageous. AWS data engineering roles commonly use services such as S3, Glue, Redshift, Lambda, Kinesis and IAM for data platforms and pipelines. … Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience ...

compute and managed data services. Strong experience with modern data warehousing/lakehouse tools (e.g., Snowflake, BigQuery, Redshift, Databricks).Hands-on orchestration experience (e.g., Airflow, Dagster, Prefect) and CI/CD for data pipelines. Experience with streaming and event-driven data (e.g., Kafka, Kinesis, Pub/Sub) is highly ...

designing and building robust, scalable, distributed data systems and pipelines, using open source and public cloud technologies. Strong experience with data orchestration tools: e.g. Apache Airflow, Dagster. Experience with big data storage and processing technologies: e.g. DBT, Spark, SQL, Athena/Trino, Redshift, Snowflake, RDBMSs (PostgreSQL/MySQL … . Knowledge of event‐driven architectures and streaming technologies: e.g. Apache Kafka, Kafka Streams, Apache Flink. Experience with public cloud environments: e.g. AWS, GCP, Azure, Terraform. Strong knowledge of software engineering practices: e.g. testing, CI/CD (Jenkins, Github Actions), agile development, git/version control, containers etc. ...

build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging technologies such as: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

Synapse/DatabricksAWS Glue/Redshift/EMRGoogle BigQuery/DataflowEnsure systems are secure, scalable, and cost-efficientData Pipeline & Workflow ManagementDevelop orchestration workflows (e.g., Airflow, Prefect, Azure Data Factory)Monitor and troubleshoot pipelines to ensure high availability and performanceImplement robust error handling, logging, and alertingData Quality & GovernanceEnsure data quality ...

basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack. You will work within a government-aligned, secure environment, ensuring data remains hosted ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT. Experience working with and designing infrastructures for large ...

release management practices. Version control and code review methodologies. Experience delivering within modern DataOps environments. Data Orchestration & Observability Experience with tools such as: Airflow Azure Data Factory (ADF) Dagster Similar data orchestration and monitoring platforms Business Intelligence Experience with: Power BI Tableau Looker Semantic modelling and metrics layer concepts ...

triaging, escalating, and driving incidents to resolution under pressure. What you’ll do: Design, develop, and maintain high-performance, product-centric data pipelines using Airflow, DBT, and Python. Architect and optimize the massive-scale data warehouse and lakehouse that serves as our single source of truth for all customer ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to 72,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to £72,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

implement highly scalable and efficient data pipelines using advanced Python (Pandas, web scraping and machine learning libraries), SQL (including advanced optimisation techniques) and Apache Airflow. Lead the extraction, transformation and loading (ETL) of large and complex datasets, including time series data, from various sources (web APIs, databases, real-time … data modelling and design skills. Extensive experience designing and building large-scale, efficient data pipelines for complex and time series data. Proven experience with Apache Airflow for data orchestration and scheduling at scale. Expertise in AWS cloud services (S3, Redshift, EC2, Lambda, etc.) is a must. Definition ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding ...

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure ...

Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity with Infrastructure as Code (Terraform/CloudFormation) Experience with orchestration tools (Airflow, dbt) Experience in financial domain #J-18808-Ljbffr ...

Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity with Infrastructure as Code (Terraform/CloudFormation) Experience with orchestration tools (Airflow, dbt) Experience in financial domain OtherLanguages English: C2 Proficient Seniority Senior London, United Kingdom of Great Britain and Northern Ireland Req. VR-124691 Strategy & Design ...

frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data Engineer, or cloud platform certifications would be an advantage. Skills: Strong ...