1 to 25 of 26 Contract Apache Airflow Jobs

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

interactive dashboards and reports using Power BI Build and optimise scalable ETL (Extract, Transform, Load) pipelines Develop and manage automated workflows using Python and Apache Airflow Manage and optimise SQL and Oracle database environments for performance, reliability, and scalability Lead end-to-end data and reporting initiatives from … with Power BI Strong Python programming skills for data processing and automation Solid experience with SQL and Oracle database systems Hands-on experience with Apache Airflow for workflow orchestration Strong understanding of ETL processes, data pipelines, and data architecture Experience managing data-focused projects including planning, coordination ...

What We're Looking For: Experience with MongoDB: Essential for this role! Technical Proficiency: Strong skills in SQL, Python, and familiarity with tools like Apache Spark, Kafka, and Airflow. Cloud Knowledge: Experience with AWS, Azure, or GCP is a plus. Data Understanding: Strong grasp of data modeling, schema design ...

clear, structured data engineering framework Build, enhance, and maintain data pipelines within Snowflake Develop transformations and data models using DBT Orchestrate workflows using Apache Airflow Write clean, modular Python code for data engineering solutions Improve pipeline reliability, performance, and observability Introduce and embed CI/CD, testing … Lead level Extensive experience with Snowflake as a core data platform Advanced SQL with strong business logic and modelling capability Experience with DBT, Airflow, and Python Experience building modular, scalable solutions within Snowflake CI/CD pipeline experience, ideally using Azure DevOps Exposure to Docker and modern software engineering ...

role Strong hands-on experience with Snowflake data platform Proficiency in SQL and data modeling techniques Experience with ETL/ELT tools (e.g., dbt, Apache Airflow, Talend, Informatica) Experience with cloud platforms (AWS, Azure, or GCP) Familiarity with data warehousing concepts and best practices Understanding of data governance ...

processes (Azure DevOps or similar) Strong communication skills and ability to work across technical and business stakeholders Nice to have Experience with orchestration tools (Airflow, Azure Data Factory or similar) Python for automation, testing or data engineering support Exposure to Databricks or modern Lakehouse environments All candidates must complete ...

secure handling of data including authentication, authorisation, encryption. Implement monitoring, alerting for large-scale distributed data workloads. Use orchestration tools such as Cloud Composer, Airflow or equivalent to operationalise pipelines. Collaborate with engineers, architects and SMEs to deliver stable, high-quality data products. Skills and Experience Strong programming skills ...

systems for unstructured data retrieval Exposure to cloud infrastructure (AWS, GCP, or Azure) for scalable data pipeline deployment Experience with modern data orchestration tools (Airflow, Prefect, dbt) What We Offer Direct access to senior football operations leadership, with genuine influence over recruitment decisions A greenfield build the opportunity ...

Snowflake best practices for data ingestion and transformation. Enhancing performance and query optimisation, ensuring cost-effective and high-performing workloads. Working with dbt and Airflow to transform raw data into structured datasets for analytical consumption. Creating clear documentation to communicate technical processes to non-technical teams. KEY SKILLS ...

expand their Devops team (multiple roles). Key requirements: Strong experience with AWS services and infrastructure as code using Terraform Data engineering expertise using Apache Spark and Python, including development in Jupyter Notebooks Workflow orchestration and containerisation experience with Apache Airflow and Docker Version control ...

prompt engineering skills and the ability to critically evaluate AI-generated code for correctness, security, and financial accuracy Desirable, but no essential: Experience with Apache Airflow or similar workflow schedulers REST or GraphQL API design experience Knowledge of fixed income, FX, or derivatives products and associated ...

Comfortable acting as a technical authority - advising on approach and best practice and communicating clearly with non-technical stakeholders Experience with Azure, SQL, Dagster, Airflow or similar tools acceptable Experience with Python scripting, React-based applications, FactSet and third-party web APIs Desirable: Understanding of financial data experience with ...

Data Engineer in secure or government environments Strong skills in Python, SQL, and data pipeline development Experience with data integration tools (e.g. Airflow, NiFi, Azure Data Factory, AWS Glue, or similar) Hands-on experience with cloud platforms (AWS, Azure, or GCP - secure tenants preferred) Knowledge of data modelling, data … environments Strong understanding of data security, governance, and access controls Desirable Experience Experience working within UK defence, intelligence, or national security programmes Exposure to Apache Kafka, Spark, Hadoop, or similar technologies Experience supporting data science or advanced analytics use cases Background working with classified datasets Contract Details Apply ...

production environment. Excellent critical thinking skills and diligence. Excellent communication skills for effective collaboration with senior engineers and stakeholders. Desirable Experience: Familiarity with Apache Airflow for workflow orchestration. Experience with Alteryx for data preparation. Knowledge of REST APIs and integration patterns. Awareness of Agile methodologies . Qualifications: Bachelor ...

production environment. Excellent critical thinking skills and diligence. Excellent communication skills for effective collaboration with senior engineers and stakeholders. Desirable Experience: Familiarity with Apache Airflow for workflow orchestration. Experience with Alteryx for data preparation. Knowledge of REST APIs and integration patterns. Awareness of Agile methodologies . Qualifications: Bachelor ...

GitHub Copilot) with significant examples of delivery improvement AWS (S3, Lambda, EC2, Glue), Snowflake, Git, CI/CD, Datadog, MS SQL Server, REST, GraphQL, Airflow (as many as possible) Degree educated or higher in a relevant technical discipline from a leading university This is a £650-700/ ...

background as a Machine Learning Engineer. Technical Skills: Strong in SQL and Python (Pandas, Scikit-learn, Jupyter, Matplotlib). Data transformation & manipulation : experience with Airflow, DBT and Kubeflow Cloud: Experience with GCP and Vertex AI (developing ML services). Expertise: Solid understanding of computer science fundamentals and time-series ...

Viya 4 Experience delivering large-scale migration programmes Desirable Knowledge of CI/CD pipelines and DevOps practices Experience integrating SAS with: o Airflow or alternative schedulers o Data platforms (eg Snowflake, Databricks) Understanding of data security and access models Experience in public sector/regulated environments (eg HMRC ...

Terraform and Ansible Awareness of critical concepts in DevOps and previous experience working in a large, structured organisation Experience working with Linux Python scripting Airflow or Packer experience would be really nice to have What you'll get in return An exciting opportunity to join an international organisation ...

including managing dependencies and build pipelines without internet access Proven background building APIs for data enrichment services Orchestration & data pipeline experience with Prefect or Airflow, and K8s or Swarm Configuration and automation of business workflow tooling (e.g., Jira) Hands-on experience with a range of databases and storage technologies ...

images to registries, and ensures secure and efficient AKS cluster deployments. Implements infrastructure as code using Terraform, including network, security, and compute resources. Supports Airflow as part of platform orchestration. Collaborates closely with Data Engineers, MLOps Engineers, and Solution Architects to ensure the infrastructure and pipelines meet performance, scalability ...

images to registries, and ensures secure and efficient AKS cluster deployments. Implements infrastructure as code using Terraform, including network, security, and compute resources. Supports Airflow as part of platform orchestration. Collaborates closely with Data Engineers, MLOps Engineers, and Solution Architects to ensure the infrastructure and pipelines meet performance, scalability ...

warehousing concepts Familiarity with version control systems, particularly Git Desirable Skills: Experience with infrastructure as code tools such as Terraform or CloudFormation Exposure to Apache Spark for distributed data processing Familiarity with workflow orchestration tools such as Airflow or AWS Step Functions Understanding of containerisation using Docker Experience ...

workloads. Practical experience working with AWS cloud services, including compute, storage, and security tooling. Infrastructure-as-code experience using Terraform. Distributed data processing using Apache Spark. Workflow scheduling and orchestration with Apache Airflow. Containerisation and environment management using Docker. CI/CD pipeline implementation and maintenance using GitLab. ...

Backend: Scala, Java (Python desirable) Frontend: HTML, CSS, JavaScript Frameworks: Play, Spark Data: Postgres, MongoDB, Hudi Cloud: AWS (S3, ECS, Aurora, Lambda) DevOps: Jenkins, Airflow, Terraform Monitoring: Kibana, Grafana Ways of working: Agile (Scrum/Kanban), CI/CD, 'you build it, you run it' What We're Looking … modern DevOps practices Experience working in Agile, multidisciplinary teams Strong analytical and problem-solving skills Clear communicator and collaborative team player Desirable Python, Docker, Airflow Big data tools (e.g. Spark, EMR) Experience with data platforms or risk/fraud systems How You'll Work Deliver value iteratively and early ...