1 to 25 of 64 Apache Airflow Jobs in the UK excluding London

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Description:Job Title Senior Airflow/Cloud Engineer (Workload Automation)Location BirminghamCorporate Title Assistant Vice PresidentService Automation provides an integrated set of management tools for IT professionals to programmatically provision and automate the operational maintenance of their hybrid application environment. It focuses on the following key capabilities:Technology Operations … responsible for availability, latency, performance, efficiency, change management, monitoring, automation, emergency response, and capacity planning for the centralized platforms of Control-M, Automic, Airflow as well as Google services Cloud Composer, Workflows and Scheduler which are utilized across the Bank for workload automation/job scheduling/batch ...

software development team Hands‐on experience designing and developing data integration/ETL pipelines from diverse data sources and formats Hands‐on experience with Apache Airflow, dbt, and Python Strong database skills with Postgres SQL, DynamoDB, Snowflake, and Databricks Experience collaborating with Agile teams, product owners, and cross … highly scalable data engineering solutions that power Moody’s next‐generation digital content platform. Design, develop, and maintain scalable data pipelines using DataBricks, Snowflake, Apache Airflow, dbt (SQL), and Python within AWS Support platform optimization, infrastructure improvements, process control enhancements, and system upgrades Collaborate with Moody’s technical ...

Data Engineer - Newcastle upon Tyne (Hybrid- 3 days onsite) - SQL, Airflow, Python, Power BI - Future Microsoft Fabric Transformation Project - Up to £55,000 Are you a Data Engineer looking for an opportunity where you can influence the direction of a growing data platform rather than simply maintain existing processes … data capabilities and embarking on an exciting data transformation journey. This role offers the chance to work hands-on with Azure SQL, Airflow, Python and Power BI while playing a key role in the organisation's future migration to Microsoft Fabric. You'll join a small, highly visible team ...

with large-scale and unstructured datasets. Python development experience, including technologies such as PySpark, Pandas and PyArrow. Expertise in distributed data processing frameworks, particularly Apache Spark. Experience building and maintaining ETL pipelines using technologies such as Apache Airflow, AWS Step Functions or Apache NiFi. Cloud platform ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

Data Engineer in secure government, defence, or intelligence settings. Strong Python and SQL development skills. Hands-on experience with data integration tools like Airflow, NiFi, Azure Data Factory, or AWS Glue. Expertise in Elasticsearch management, indexing, and query optimisation. Knowledge of data modelling, warehousing, and modern data platform architectures. ...

Glue, ECR, EC2, EMR, DynamoDB, Lake Formation Experience with Infrastructure as Code (Terraform) Proficiency in Python, SQL, and familiarity with PySpark Experience with Apache Spark and data processing frameworks Hands-on experience with Apache Airflow for workflow orchestration Experience using Jupyter Notebooks and/or Amazon Athena ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

C#, Golang, Java, Kotlin, TypeScript, or Rust Data Platforms: BigQuery, Snowflake, Redshift, Databricks, PostgreSQL, MySQL, MongoDB, or other SQL/NoSQL technologies Data Tooling: Airflow, dbt, AWS Glue, Step Functions, Spark, Hadoop, or similar tooling Infrastructure: AWS, Terraform, Docker, Kubernetes, or other cloud/platform technologies CI/ ...

Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills with ...

experience in Data Engineering with a focus on ETL/ELT development. Strong SQL and Python programming skills. Experience with DBT, Git, Terraform, IAM, Airflow/Composer, and Docker. Experience working with cloud platforms, preferably Google Cloud Platform (BigQuery), with Azure or AWS also considered. Knowledge of CI/ ...

large-scale data platforms and end-to-end technical solutions within enterprise environments. Strong hands-on experience with open-source data technologies such as Apache Spark, Trino, Airflow, Apache Superset , or similar platforms. Demonstrable security architecture expertise, including threat modelling, IAM (RBAC/ABAC), encryption, secrets management ...

need: Strong Python, PySpark, SQL and Databricks or Snowflake experience AWS cloud data engineering expertise Experience building scalable batch and real-time data pipelines Airflow, CI/CD, Git and modern engineering practices Knowledge of Terraform or similar IaC tools Experience with Docker and Kubernetes is beneficial Exposure ...

Engineering with a strong focus on ETL/ELT development. Strong proficiency in SQL and Python. Hands-on experience with DBT, Git, Terraform, IAM, Airflow/Composer, and Docker. Cloud platform experience, preferably Google Cloud Platform (BigQuery), with Azure or AWS also considered. Experience building and maintaining modern data ...

with large datasets. Hands-on experience with AWS services and cloud-based data platforms. Experience building and maintaining production-grade data pipelines. Experience using Airflow for workflow orchestration. Strong version control experience using Git. Understanding of data modelling principles and database design. Excellent problem-solving and analytical skills. Strong ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

maintaining scalable data pipelines and integrations. Developing robust ETL and data transformation processes to support analytics and reporting requirements. Working with Snowflake, DBT, Apache Airflow, AWS, SQL and Python to deliver high-quality data solutions. Collaborating with Data Architecture, DevOps and business stakeholders to ensure trusted and accessible … ideally within AWS. Data modelling, transformation and orchestration methodologies. Delivering high-quality, well-documented and thoroughly tested data solutions. Experience with Snowflake, DBT and Apache Airflow. Interest in AI and emerging technologies within the data engineering landscape. THE BENEFITS: You will receive a salary ...

/ELT, and various database technologies. Technical Versatility: Fluency in data processing languages (Python, SQL, PySpark) and openness to tech stacks such as SSIS, Airflow, Spark, Databricks, Fabric. Cloud Capability: Experience with at least one major cloud provider (AWS, Azure, or Google Cloud) and curiosity to learn others. Bridging ...

Python skills * Hands-on experience designing and developing ETL pipelines and data workflows * Experience designing and maintaining data pipelines using technologies such as Apache Airflow or Spark * Experience working with cloud platforms including AWS, Azure or GCP * Strong understanding of data modelling, integration and governance principles * Experience working ...

distributed processing frameworks like SparkSQL Experience with CI/CD, version control (e.g. Git, Jenkins) and pipeline orchestration tools such as Airflow Due to the volume of applications received for positions, it will not be possible to respond to all applications and only applicants who are considered suitable ...

within multi-disciplinary teams, delivering modern data platforms into large organisations. Previous practical experience of using both SQL and task orchestration software (eg. dbt, Airflow/Prefect/Dagster or equivalent) to develop production-grade pipelines that bring data to life Strong Snowflake experience is preferred, and Databricks experience ...

implementing agentic guardrails. Data Engineering (30%): Practical experience implementing components of data pipelines, including real-time streaming tools (AWS Kinesis, Kafka), data orchestration (dbt, Airflow), and managing vector databases for RAG architectures. Observability & Cost Management: SRE/Platform experience with the practical application of real-time monitoring and cloud ...