14 of 14 Contract Apache Airflow Jobs

professionals with strong Python development expertise, cloud engineering experience, and exposure to modern DevOps and data engineering practices. Must Have: AWS services; Terraform; Apache Spark; Python; Apache Airflow; Docker; GitLab; Security scanning agent; Jupyter; Notebooks; Key Responsibilities Design, develop, and maintain scalable Python applications and services Build … optimize cloud-native data processing pipelines Develop and manage workflows using orchestration tools such as Airflow Work with large-scale data processing frameworks including Apache Spark Deploy and maintain containerized applications using Docker Implement infrastructure automation using Terraform Integrate solutions with enterprise data and secure file transfer systems ...

data validation and application testing processes across distributed systems Investigate production issues using data analysis to identify root causes Work with orchestration tools (e.g., Apache Airflow) to monitor workflows and troubleshoot failures Validate data using tools such as Jupyter Notebooks and Amazon Athena Analyse data across multiple formats … additional programming language (e.g., Java) Experience with scripting languages such as Bash Strong understanding of automated testing frameworks and methodologies Hands-on experience with Apache Airflow and log analysis Familiarity with data processing tools such as Apache Spark Experience working with cloud platforms and services, including: Monitoring ...

Data Engineer - GCP, BigQuery, Airflow Location: London (Hybrid) Contract: 12 Months Rate: £350 per day PAYE (Inside IR35) Start Date: ASAP A leading technology and data-driven organisation is seeking an experienced Data Engineer to join its data engineering team on an initial 12-month contract. The successful candidate … across critical business data platforms. Key Responsibilities Build, maintain, and optimise scalable data pipelines within GCP Develop and manage workflows using Cloud Composer and Apache Airflow Design and support data solutions using BigQuery Implement data quality checks and monitoring frameworks Improve observability and operational performance of data platforms ...

architecture and end-to-end system design Proven Experience with big data platforms at scale Deep knowledge of open-source data ecosystems, such as: Apache Spark Trino Airflow Apache Superset Cloud & Infrastructure Experience across cloud and on-premise environments Familiarity with AWS, GCP, and/or Azure ...

Experience Extensive experience in data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience … with modern data warehousing solutions (e.g., Snowflake, BigQuery, Redshift) Familiarity with orchestration tools (e.g., Airflow, Databricks Workflows) Strong understanding of data modelling and lakehouse architecture Databricks certifications (e.g., Databricks Certified Data Engineer) Experience with streaming data pipelines (Kafka, Structured Streaming) Experience with CI/CD and DevOps practices Knowledge ...

reviews to maintain high engineering standards. As a Data Engineer you will have the following key skills Cloud: GCP Warehousing: BigQuery/Snowflake Orchestration: Airflow Backend: Python, SQL, Shell Languages: Scala/Java DevOps: Terraform, Docker, Kubernetes Observability: Monte Carlo BI: Looker AI/ML: MLOps London | 12 Months ...

Snowflake Toolkit: Hands-on with Snowpipe, Streams, Tasks, Dynamic Tables, and Stored Procedures. Nice to Have: Exposure to DBT, Snowpark, Snowflake AI/ML, Airflow, and Snowflake administration (RBAC, warehouse sizing). Mindset: Strong problem-solving abilities with an ownership mindset for end-to-end delivery. This ...

/SD/PP etc.) required for December cutover. Build scalable ingestion pipelines using Delta Lake/Parquet and modern orchestration tools (ADF, dbx, Airflow, etc.). Collaborate with SAP functional teams to validate business logic, deltas, and data quality. Implement CI/CD and DevOps practices for data ...

preferred) or similar platforms: Snowflake, BigQuery, Redshift, Synapse, or equivalent Strong proficiency with: SQL (advanced level) Spark/PySpark Data pipeline orchestration tools (e.g., Airflow, ADF, etc.) Data Modelling Extensive experience designing: Fact and dimension tables Star and snowflake schemas Aggregation layers and semantic models Data Migration & Mapping Demonstrated ...

Python, Bash, Ansible, Puppet) Familiarity with containerisation (Docker, Kubernetes) Nice to have: Experience with CI/CD pipelines (GitLab, Jenkins) Exposure to data tooling (Airflow, NiFi, Elasticsearch) Experience with Cloudera/Hadoop ecosystems (Spark, Hive, HDFS) Why join? Work on modern data platforms at scale Collaborate with data scientists ...

Ensure solutions meet security, compliance, and regulatory standards Key Skills Required: Experience in solution design/architecture Strong background in big data tools (Spark, Airflow, Trino, etc.) Knowledge of cloud platforms (AWS/Azure/GCP) Understanding of cybersecurity principles (IAM, encryption, secure design) Ability to work hands ...

This is a great opportunity to contribute greatly to an extremely engineering-focused organization and further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform, Google Cloud Platform, etc; You will own engineering efforts to provide partner teams within the organisation with data sources that build … opportunity for the role to go permanent. What You'll Do: Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow; Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality ...

focus areas: Building scalable data pipelines using Apache Spark & Airflow Working within a cloud-based, containerised environment (Kubernetes) Contributing to a wider data platform modernisation programme Supporting both batch and streaming workloads Tech: Python, Java, Spark, Airflow, SQL, Cloud (AWS/Azure), Kubernetes (Kafka/streaming nice ...

within air-gapped environments, ensuring secure deployment of ML solutions Collaborate with data scientists to operationalise ML models Utilise tools such as MLflow and Airflow for model tracking and orchestration Support cloud-based deployments across AWS and Google Cloud Platform Ensure performance, reliability, and scalability of machine learning systems … Experience building CI/CD pipelines using GitLab (YAML) Understanding of DevSecOps principles and tooling Exposure to ML lifecycle tools such as MLflow and Airflow Ability to work in highly secure or air-gapped environments Strong problem-solving and collaboration skills What you'll get in return Opportunity ...