24 of 24 Contract Apache Airflow Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

interactive dashboards and reports using Power BI Build and optimise scalable ETL (Extract, Transform, Load) pipelines Develop and manage automated workflows using Python and Apache Airflow Manage and optimise SQL and Oracle database environments for performance, reliability, and scalability Lead end-to-end data and reporting initiatives from … with Power BI Strong Python programming skills for data processing and automation Solid experience with SQL and Oracle database systems Hands-on experience with Apache Airflow for workflow orchestration Strong understanding of ETL processes, data pipelines, and data architecture Experience managing data-focused projects including planning, coordination ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Notebooks for data exploration and development Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

Design, build, and maintain cloud-based infrastructure using AWS Develop and manage Infrastructure as Code using Terraform Support and optimise data processing pipelines using Apache Spark and Airflow Collaborate with cross-functional teams to deliver secure and scalable solutions Implement CI/CD pipelines using GitLab Containerise applications … Access Layer Secure File Transfer (SFT) Key Skills & Experience Strong experience with AWS services Expertise in Terraform (Infrastructure as Code) Hands-on experience with: Apache Spark Python Apache Airflow Docker GitLab CI/CD Experience with security scanning tools such as: Trivy Trend Micro Wiz (or similar ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema design ...

clear, structured data engineering framework Build, enhance, and maintain data pipelines within Snowflake Develop transformations and data models using DBT Orchestrate workflows using Apache Airflow Write clean, modular Python code for data engineering solutions Improve pipeline reliability, performance, and observability Introduce and embed CI/CD, testing … Lead level Extensive experience with Snowflake as a core data platform Advanced SQL with strong business logic and modelling capability Experience with DBT, Airflow, and Python Experience building modular, scalable solutions within Snowflake CI/CD pipeline experience, ideally using Azure DevOps Exposure to Docker and modern software engineering ...

role Strong hands-on experience with Snowflake data platform Proficiency in SQL and data modeling techniques Experience with ETL/ELT tools (e.g., dbt, Apache Airflow, Talend, Informatica) Experience with cloud platforms (AWS, Azure, or GCP) Familiarity with data warehousing concepts and best practices Understanding of data governance ...

systems for unstructured data retrieval Exposure to cloud infrastructure (AWS, GCP, or Azure) for scalable data pipeline deployment Experience with modern data orchestration tools (Airflow, Prefect, dbt) What We Offer Direct access to senior football operations leadership, with genuine influence over recruitment decisions A greenfield build the opportunity ...

secure handling of data including authentication, authorisation, encryption. Implement monitoring, alerting for large-scale distributed data workloads. Use orchestration tools such as Cloud Composer, Airflow or equivalent to operationalise pipelines. Collaborate with engineers, architects and SMEs to deliver stable, high-quality data products. Skills and Experience Strong programming skills ...

environments Solid experience with SQL and data modelling concepts Strong analytical and problem-solving skills Nice to Have Skills Experience with data pipeline tools (Airflow, Kafka, Spark) Knowledge of cloud platforms (AWS, Azure or GCP) Familiarity with relational and NoSQL databases Additional programming experience (Python, Scala) Essential Requirements Active ...

pipelines/services, that are build and find ways to improve system robustness, resilience and stability Key skills required: Understanding of data processing using Apache Spark Use of Python, SQL, and familiarity with PySpark Experience using Apache Airflow for task orchestration Understanding of EMR and reviewing output ...

prompt engineering skills and the ability to critically evaluate AI-generated code for correctness, security, and financial accuracy Desirable, but no essential: Experience with Apache Airflow or similar workflow schedulers REST or GraphQL API design experience Knowledge of fixed income, FX, or derivatives products and associated ...

background as a Machine Learning Engineer. Technical Skills: Strong in SQL and Python (Pandas, Scikit-learn, Jupyter, Matplotlib). Data transformation & manipulation : experience with Airflow, DBT and Kubeflow Cloud: Experience with GCP and Vertex AI (developing ML services). Expertise: Solid understanding of computer science fundamentals and time-series ...

Viya 4 Experience delivering large-scale migration programmes Desirable Knowledge of CI/CD pipelines and DevOps practices Experience integrating SAS with: o Airflow or alternative schedulers o Data platforms (eg Snowflake, Databricks) Understanding of data security and access models Experience in public sector/regulated environments (eg HMRC ...

Data Engineer in secure or government environments Strong skills in Python, SQL, and data pipeline development Experience with data integration tools (e.g. Airflow, NiFi, Azure Data Factory, AWS Glue, or similar) Hands-on experience with cloud platforms (AWS, Azure, or GCP - secure tenants preferred) Knowledge of data modelling, data … environments Strong understanding of data security, governance, and access controls Desirable Experience Experience working within UK defence, intelligence, or national security programmes Exposure to Apache Kafka, Spark, Hadoop, or similar technologies Experience supporting data science or advanced analytics use cases Background working with classified datasets Contract Details Apply ...

production environment. Excellent critical thinking skills and diligence. Excellent communication skills for effective collaboration with senior engineers and stakeholders. Desirable Experience: Familiarity with Apache Airflow for workflow orchestration. Experience with Alteryx for data preparation. Knowledge of REST APIs and integration patterns. Awareness of Agile methodologies . Qualifications: Bachelor ...

production environment. Excellent critical thinking skills and diligence. Excellent communication skills for effective collaboration with senior engineers and stakeholders. Desirable Experience: Familiarity with Apache Airflow for workflow orchestration. Experience with Alteryx for data preparation. Knowledge of REST APIs and integration patterns. Awareness of Agile methodologies . Qualifications: Bachelor ...

GitHub Copilot) with significant examples of delivery improvement AWS (S3, Lambda, EC2, Glue), Snowflake, Git, CI/CD, Datadog, MS SQL Server, REST, GraphQL, Airflow (as many as possible) Degree educated or higher in a relevant technical discipline from a leading university This is a £650-700/ ...

Comfortable acting as a technical authority - advising on approach and best practice and communicating clearly with non-technical stakeholders Experience with Azure, SQL, Dagster, Airflow or similar tools acceptable Experience with Python scripting, React-based applications, FactSet and third-party web APIs Desirable: Understanding of financial data experience with ...

images to registries, and ensures secure and efficient AKS cluster deployments. Implements infrastructure as code using Terraform, including network, security, and compute resources. Supports Airflow as part of platform orchestration. Collaborates closely with Data Engineers, MLOps Engineers, and Solution Architects to ensure the infrastructure and pipelines meet performance, scalability ...

Robert Half Technology are assisting a rapidly growing sports fashion brand to recruit a Data Engineer on a contract basis. This is an exciting opportunity to join a fast-paced SME environment at a pivotal ...

workloads. Practical experience working with AWS cloud services, including compute, storage, and security tooling. Infrastructure-as-code experience using Terraform. Distributed data processing using Apache Spark. Workflow scheduling and orchestration with Apache Airflow. Containerisation and environment management using Docker. CI/CD pipeline implementation and maintenance using GitLab. ...