1 to 25 of 142 Remote/Hybrid Apache Airflow Jobs in England

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

Consultant to support the implementation and deployment of Astronomer (Astro) within a leading asset management organisation. Job Responsibilities/Objectives Astronomer is built around Apache Airflow and is used to develop, schedule, automate and manage complex data pipelines. The organisation is currently implementing the platform and requires … ownership of the implementation and providing training and support to upskill the wider team. Experience with Astronomer/Astro is essential, alongside knowledge of Apache Airflow, dbt and Snowflake. Key Responsibilities Lead the implementation, build and deployment of Astronomer (Astro). Configure and manage Apache Airflow ...

seeking a Technical Delivery Lead to drive the engineering and technical strategy for a scalable, multi-tenant data integration platform built on Apache Airflow and Google Cloud. In this role, you will lead the delivery of production-grade data pipelines and cloud solutions, working closely with cross-functional … guiding teams in a collaborative environment. Roles and Responsibilities Lead the engineering and technical strategy for a scalable multi‐tenant data integration platform leveraging Apache Airflow and GCP technologies. Build and maintain production‐grade ETL/ELT pipelines orchestrated with Apache Airflow. Utilize container platforms such ...

practices. Responsibilities will include: Providing Staff-level technical leadership across complex data engineering and modernisation initiatives. Remaining deeply hands‐on with Python, PySpark and Apache Spark , independently building and proving engineering solutions. Designing and optimising scalable data pipelines and processing frameworks across AWS . Tackling complex Spark performance challenges … including shuffles, joins, data skew, partitioning and memory optimisation. Designing and challenging engineering approaches across AWS Glue, Apache Airflow and associated technologies. Reviewing existing code, identifying performance bottlenecks and driving improvements in engineering quality. Establishing reusable engineering patterns, frameworks and approaches to accelerate large‐scale data migrations. Applying ...

Data Engineer — AWS, Python & Airflow Rate: £550 per day, outside IR35 Location: Remote Duration: Initial 6-month contract, with a further 12-month extension planned Clearance: Active SC clearance, used within the last 12 months Nationality: Sole British nationals only; dual citizenship is not permitted The opportunity … urgently looking for an experienced Data Engineer with strong AWS, Python and Apache Airflow skills to join a programme on an initial six-month contract. You'll help build and maintain reliable data pipelines, develop cloud-based data solutions and make data available for downstream analytics and programme ...

organisation. Responsibilities: Build scalable ETL and ELT pipelines within AWS Develop and optimise data models in Amazon Redshift Create and manage workflows using Apache Airflow Integrate NetSuite and other Finance data sources Establish trusted data-quality, monitoring and validation processes Own solutions from initial requirements through to production … Experience Required: Strong commercial Python and SQL experience Hands‐on experience with AWS data services Amazon Redshift or comparable cloud data‐warehouse experience Apache Airflow or similar orchestration experience Strong ETL/ELT, data‐modelling and production pipeline knowledge The confidence to work directly with Finance and technical ...

Knowledge of: MLOps Feature Engineering Exposure to data analysis and ML tools such as: scikit-learn pandas NumPy XGBoost LightGBM Familiarity with orchestration tools: Apache Oozie Apache Airflow Why you’ll love it here We want to make Prima a happy and empowering place to work. ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

Experience of working in an Agile delivery team Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

platforms:AWS,Azureand GCP Modern data platforms:Databricks, Snowflake and cloud-native data services Data architecture:Lakehouse, medallion, datameshand domain-oriented patterns Data processing:Apache Spark,SQLand distributed processing technologies Data storage:Delta Lake, ApacheIcebergand open table formats Data pipelines:Enterprise-scale ETL/ELT,batchand streaming patterns Transformation … quality:dbt, Great Expectations and equivalent tooling Orchestration:Apache Airflow and cloud-native orchestration services Event streaming:Kafka and cloud-native event-streaming technologies Data modelling:Dimensional,analyticaland domain-oriented modelling Data governance:Catalogue, metadata, lineage, quality, security,privacyand access controls Data products:Reusable, discoverable and well-governed ...

this clearance as a condition of employment. Nice to have Experience with distributed data processing or streaming tools (e.g. Spark, Kafka) Workflow orchestration (e.g. Airflow, Dagster or similar) Working with LLMs, vector databases, embeddings or RAG patterns MLOps practices: model versioning, deployment and monitoring Containers and infrastructure as code ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

required, we are seeking Engineers who have experience with one or more of the following technological areas: Analytics & Platform Frameworks Orchestration like Airflow or AWS Glue Database Technologies Data processing libraries & Schema Management tools like Pandas and Pydantic Storage Formats: Familiarity with data types such as JSON and Parquet ...

including supporting automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

automated deployments, infrastructure management, monitoring, and scaling Coaching and mentoring junior engineers and promoting engineering best practices Skills & Experience: Understanding of data processing using Apache Spark and Apache Airflow for task orchestration Use of Python, SQL, and familiarity with PySpark Understanding of EMR and reviewing output logs ...

production pipelines Strong SQL and Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice ...

solving and communication skills. Able to work independently and remotely, and to travel to Newcastle once a week. Desirable skills and experience Experience with Apache Spark, Databricks or Kafka. Familiarity with orchestration tools such as Airflow. Experience with CI/CD for data pipelines and Infrastructure as Code. Knowledge ...

with modern cloud data warehouses, ideally Google BigQuery. Practical understanding of dimensional data modelling principles .### Experienceusing tools in our tech stack such as Airflow, Terraform, or Grafana. Previous### experienceworking within Agile delivery teams .### BenefitsWe believe in supporting our people with meaningful, flexible### benefitsthat prioritise well-being ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working ...

Python and SQL programming skills, with experience developing and maintaining data pipelines • Familiarity with ETL/ELT technologies and data streaming tools such as Airflow, Azure, SSIS, Spark or Kafka • Experience working with databases and data platforms such as PostgreSQL, SQL Server, Oracle, Snowflake or similar technologies • Knowledge ...

analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases, and can integrate the relevant data quality checks ...

need as the SeniorData Engineer: Passionfor building scalable, resilient cloud data platforms Strongexperience with Databricks or Snowflake on AWS ProvenPython skills, including Spark and Airflow expertise AdvancedSQL skills and end‐to‐end data modelling experience Experiencebuilding batch and real-time data integrations Hands-onCICD skills with Git, Jenkins ...

they can be applied to improve quality processes and drive continuous improvement. Exposure to data engineering and analytics tools (e.g., Databricks, Spark, Airflow or dbt). Awareness of cloud platforms (AWS, Azure or GCP). Experience with Git, CI/CD or Agile practices (from university or projects). ...

SQL. Well-versed in software engineering and systems design principles. Hands-on experience with Azure cloud data platforms is essential. Familiarity with Spark and Apache Airflow is required. Proficient in Linux/Unix/*nix environments. Strong knowledge of ETL tools and data modelling techniques. Candidates will need ...