26 to 50 of 355 Apache Airflow Jobs in England

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business … Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g., dbt) Experience designing and operating orchestration pipelines using Airflow or similar tools Experience designing and building streaming pipelines using Kafka, Pub/Sub, or similar messaging systems Preferred qualifications, capabilities, and skills Data modeling ...

Amazon Redshift AWS Lambda Amazon Athena Amazon RDS AWS IAM Amazon CloudWatch Experience with services such as Kinesis, EMR, Step Functions or MWAA/Airflow would be advantageous. AWS data engineering roles commonly use services such as S3, Glue, Redshift, Lambda, Kinesis and IAM for data platforms and pipelines. … Automated deployments across environments. Monitoring and logging. Release and configuration management. Production support and incident troubleshooting. Python or another scripting language. AWS certification. Terraform. Apache Airflow/MWAA. Databricks or Spark. Kafka/Kinesis. dbt. Experience working in Agile/Scrum environments. Experience 4+ years of experience ...

build, and maintain scalable and robust data pipelines using SQL and Python, and leveraging technologies such as: Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow and Pyspark. Lead the integration of complex data systems and ensure consistency and accuracy of data across multiple platforms. Implement continuous integration ...

Snowflake, BigQuery, Redshift, EMR, or similar). Our stack is Databricks and Snowflake; strong equivalent experience elsewhere transfers. Experience with a workflow orchestrator (e.g. Airflow, Dagster, or Databricks Workflows). Proven system and platform design judgment: you own architectural decisions and can defend the trade-offs. Greenfield, scaling ...

basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack. You will work within a government-aligned, secure environment, ensuring data remains hosted ...

stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres, Gradle, BERT. Experience working with and designing infrastructures for large ...

triaging, escalating, and driving incidents to resolution under pressure. What you’ll do: Design, develop, and maintain high-performance, product-centric data pipelines using Airflow, DBT, and Python. Architect and optimize the massive-scale data warehouse and lakehouse that serves as our single source of truth for all customer ...

Senior Data Engineer - Python, SQL, Airflow, Azure, Pyspark, Kubernetes Salary: Up to £72,000 per annum Benefits: Excellent benefits package including annual bonus, free onsite gym, and company pension scheme. About the Role We are seeking an experienced Senior Data Engineer to join an innovative SaaS business that … Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

principles. Comfortable working within Agile delivery environments. Experience with CI/CD pipelines, ideally using Azure DevOps or similar. Nice to Have Experience with Apache Airflow or dbt. Exposure to BigQuery/GCP. Azure or other cloud platform certifications Knowledge of data governance and data management frameworks Ideal ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python . Experience with data engineering and pipeline technologies such as Apache Airflow and Spark . Experience working with cloud platforms, particularly AWS and Azure . Knowledge of big data technologies and distributed data processing. ...

supported by solid software engineering fundamentals (version control, testing, CI/CD). Experience building and operating data pipelines using workflow orchestration frameworks (e.g. Apache Airflow), with a focus on reliability, observability, dependency management, and operational resilience. Experience designing and operating cloud-native data platforms (AWS or Azure ...

Data Engineer Associate) Experience with streaming data (Kinesis, MSK/Kafka) Familiarity with Infrastructure as Code (Terraform/CloudFormation) Experience with orchestration tools (Airflow, dbt) Experience in financial domain #J-18808-Ljbffr ...

frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data Engineer, or cloud platform certifications would be an advantage. Skills: Strong ...

analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

batch and real-time pipelines Work with streaming and event-driven technologies such as Kafka, Flink, Kinesis, and Pub/Sub Orchestrate workflows using Airflow, Dataflow, and Glue Process and transform large datasets using Spark and Flink Design production-ready systems Work across relational, NoSQL, and analytical data stores ...

Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills with ...

analytics infrastructure. Set engineering standards, tooling decisions, and technical direction across the data stack (Python, SQL, DBT, AWS: S3, Glue, EMR, Lambda, Aurora, Airflow, Terraform). Ensure data availability, integrity, and performance as the platform scales globally. Drive best practices in data quality, automated testing, anomaly detection ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

data engineering, with demonstrated expertise in designing and implementing scalable, flexible modern data architectures Experience building modern data lake/lakehouse architectures (e.g., Apache Iceberg) in cloud environments Strong knowledge of AWS (S3, Redshift) or Azure data services Experience with ETL/ELT processes and data ingestion Proven expertise … using transformation tools such as dbt Experience with data pipeline orchestration tools (such as Apache Airflow) Experience with best practices applied to data (e.g., testing, CI/CD) Advantageous Experience developing dashboards and reporting solutions using data visualization tools such as Power BI or Amazon QuickSight Experience with ...

Cloud Solutions with strong expertise in Google Cloud Platform (GCP). Expert-level hands-on experience with GCP data services including BigQuery, Cloud Composer (Airflow), Dataflow, Dataproc, Cloud Storage, Pub/Sub, and Vertex AI. Advanced proficiency in dbt, Python, and SQL for designing, building, and optimizing cloud-native ...