26 to 50 of 367 Apache Airflow Jobs in England

including supporting automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

production pipelines Strong SQL and Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice ...

solving and communication skills. Able to work independently and remotely, and to travel to Newcastle once a week. Desirable skills and experience Experience with Apache Spark, Databricks or Kafka. Familiarity with orchestration tools such as Airflow. Experience with CI/CD for data pipelines and Infrastructure as Code. Knowledge ...

with modern cloud data warehouses, ideally Google BigQuery. Practical understanding of dimensional data modelling principles .### Experienceusing tools in our tech stack such as Airflow, Terraform, or Grafana. Previous### experienceworking within Agile delivery teams .### BenefitsWe believe in supporting our people with meaningful, flexible### benefitsthat prioritise well-being ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

Python and SQL programming skills, with experience developing and maintaining data pipelines • Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka • Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies • Knowledge ...

multiple data sources Distributed computing for data intensive work such as Spark and Databricks Cloud platforms including AWS, Azure and GCP Orchestration tools like Airflow Modern DevOps practices including version control, CI/CD, and infrastructure as code using Terraform or Kubernetes Experience building scalable backend applications Understanding ...

optimise batch and real-time pipelines Work with streaming and event-driven tech such as Kafka, Flink, Kinesis, Pub/Sub Orchestrate workflows using Airflow, Dataflow, Glue Work at Scale Process and transform large datasets using Spark and Flink Design systems that perform in production - not just on paper ...

analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

operational issues. Understanding of platform security, resilience, scalability and availability. Experience working within Agile and/or DevOps delivery environments . Desirable: Helm, ArgoCD, Airflow, dbt, Bash, Ansible, Packer, Elasticsearch/OpenSearch, Prometheus, Grafana, ELK or Datadog. Experience working within public sector, utilities or other regulated environments is advantageous. ...

translate them into a practical, maintainable data solution Strong problem‐solving and analytical skills Orchestration and monitoring tools, e.g. Databricks Workflows, dbt or Airflow (desirable) Integrating SaaS platforms through APIs, e.g. Salesforce or Workday (desirable) Building data catalogues, dictionaries or certified datasets, e.g. Unity Catalog, Microsoft Purview or Power ...

adhere to the D.R.Y principle, minimising manual intervention and promote automation. We're looking for these essential skills Hands‐on experience with DBT and Apache Airflow, building scalable data transformations and DAGs to orchestrate complex, reliable data pipelines. Manage task dependencies, automated retries and error handling to ensure ...

Athena, Lambda, S3, Glue and Step Functions, but we're open to transferable experience solving similar problems. Experience with Snowflake, Spark, Scala or Airflow, or transferable experience with comparable technologies and solving similar data engineering challenges. Experience with Git, GitHub and CI/CD practices, including tools such ...

production-quality ETL/ELT pipelines and automated data workflows. Experience integrating heterogeneous data sources, databases and APIs. Experience with workflow orchestration tooling (e.g. Airflow, Dagster or dbt) for scheduling and operating production data pipelines. Good understanding of data quality, metadata, lineage, testing and reproducibility. Ability to evaluate technical ...

/CD, version control, and testing practices Strong problem-solving and communication skills Desirable Skills: Experience with tools such as Databricks, Data Factory, Fabric, Airflow, DBT, or equivalent Experience working with structured and semi-structured data sources Understanding of data security, governance, and compliance requirements Our culture and benefits ...

warehousing, data pipelines and data transformation (extract, transform, and load) processes and supporting technologies such as Google Dataflow, DBT, EMR, CI/CD Pipelines, Airflow DAGs, and other analytics tools. Experience with cloud-based data infrastructures (Ideally GCP, but AWS or Azure would also suffice). Expertise in managing ...

Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow). Strong experience with data engineering concepts (ETL/ELT, data integration, warehousing, vector databases/search) A little bit more about us... ...

need as the SeniorData Engineer: Passionfor building scalable, resilient cloud data platforms Strongexperience with Databricks or Snowflake on AWS ProvenPython skills, including Spark and Airflow expertise AdvancedSQL skills and end‐to‐end data modelling experience Experiencebuilding batch and real-time data integrations Hands-onCICD skills with Git, Jenkins ...

Spark, Databricks, Hadoop or cloud-native alternatives).* Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow).* Strong experience with data engineering concepts (ETL/ELT, data integration, warehousing, vector databases/search)**A little bit more about us... ...

they can be applied to improve quality processes and drive continuous improvement. Exposure to data engineering and analytics tools (e.g., Databricks, Spark, Airflow or dbt). Awareness of cloud platforms (AWS, Azure or GCP). Experience with Git, CI/CD or Agile practices (from university or projects). ...

software engineering experience using Python and/or other modern programming languages. Experience developing and orchestrating data pipelines using technologies such as DBT, Apache Spark, Airflow, Dagster, Kafka, Flink, or equivalent tooling. Solid understanding of data lakehouse, warehouse, and medallion architecture patterns. Experience applying DevOps and DataOps best ...

adhere to the D.R.Y principle, minimising manual intervention and promote automation. **We're looking for these essential skills** * Hands-on experience with DBT and Apache Airflow, building scalable data transformations and DAGs to orchestrate complex, reliable data pipelines.* Manage task dependencies, automated retries and error handling to ensure ...

SQL. Well-versed in software engineering and systems design principles. Hands-on experience with Azure cloud data platforms is essential. Familiarity with Spark and Apache Airflow is required. Proficient in Linux/Unix/*nix environments. Strong knowledge of ETL tools and data modelling techniques. Candidates will need ...