101 to 125 of 286 Apache Airflow Jobs

systems (e.g. Git) Desirables: Experience with cloud-based ML infrastructure, particularly GCP (Vertex AI, BigQuery), or equivalent (e.g. AWS, Azure) Exposure to orchestration tools such as Kubeflow pipelines or Airflow Familiarity with DBT or similar tools for modelling data in data warehouses Desire to build interpretable and explainable ML models (using techniques such as SHAP) Desire to quantify the More ❯

systems (e.g. Git) Desirables: Experience with cloud-based ML infrastructure, particularly GCP (Vertex AI, BigQuery), or equivalent (e.g. AWS, Azure) Exposure to orchestration tools such as Kubeflow pipelines or Airflow Familiarity with DBT or similar tools for modelling data in data warehouses Desire to build interpretable and explainable ML models (using techniques such as SHAP) Desire to quantify the More ❯

systems (e.g. Git) Desirables: Experience with cloud-based ML infrastructure, particularly GCP (Vertex AI, BigQuery), or equivalent (e.g. AWS, Azure) Exposure to orchestration tools such as Kubeflow pipelines or Airflow Familiarity with DBT or similar tools for modelling data in data warehouses Desire to build interpretable and explainable ML models (using techniques such as SHAP) Desire to quantify the More ❯

In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure cloud platform. You will leverage your expertise in PySpark, Apache Spark, and Apache Airflow to process and orchestrate large-scale data workloads, ensuring data quality, efficiency, and scalability. If you have a passion for data engineering and … significant impact, we encourage you to apply! Job Responsibilities ETL/ELT Pipeline Development: Design, develop, and optimize efficient and scalable ETL/ELT pipelines using Python, PySpark, and Apache Airflow. Implement batch and real-time data processing solutions using Apache Spark. Ensure data quality, governance, and security throughout the data lifecycle. Cloud Data Engineering: Manage and optimize … effectiveness. Implement and maintain CI/CD pipelines for data workflows to ensure smooth and reliable deployments. Big Data & Analytics: Develop and optimize large-scale data processing pipelines using Apache Spark and PySpark. Implement data partitioning, caching, and performance tuning techniques to enhance Spark-based workloads. Work with diverse data formats (structured and unstructured) to support advanced analytics and More ❯

In this role, you will be responsible for designing, building, and maintaining robust data pipelines and infrastructure on the Azure cloud platform. You will leverage your expertise in PySpark, Apache Spark, and Apache Airflow to process and orchestrate large-scale data workloads, ensuring data quality, efficiency, and scalability. If you have a passion for data engineering and … desire to make a significant impact, we encourage you to apply! Job Responsibilities Data Engineering & Data Pipeline Development Design, develop, and optimize scalable DATA workflows using Python, PySpark, and Airflow Implement real-time and batch data processing using Spark Enforce best practices for data quality, governance, and security throughout the data lifecycle Ensure data availability, reliability and performance through … data processing workloads Implement CI/CD pipelines for data workflows to ensure smooth and reliable deployments. Big Data & Analytics: Build and optimize large-scale data processing pipelines using Apache Spark and PySpark Implement data partitioning, caching, and performance tuning for Spark-based workloads. Work with diverse data formats (structured and unstructured) to support advanced analytics and machine learning More ❯

required GCP Focus What will make you stand out: Proven experience with data warehousing and ETL/ELT. Google BigQuery, GCP, SQL etc. Experience with Python, TypeScript etc. Experience Apache Airflow, Cloud Composer etc. Experience with stream processing (e.g. Dataflow, Pub/Sub). Experience with data visualization tools (e.g. Looker Studio, Tableau). This role is an More ❯

in New York, NY Pay Range: $95,493 $190,962 Job Duties: Improve and extend software systems that speed up data product development by providing automation and abstraction. Maintain Apache Airflow and Airbyte for creation of data pipelines and orchestration. Utilize container technologies such as Docker and Kubernetes. Develop and maintain a suite of internal tools to facilitate … equivalent, plus two (2) years of experience in the job offered or a related occupation. Requires 2 years of experience in each: Container technologies, e.g. Helm, Docker or Kubernetes. Apache Airflow Observability of data platforms (Datadog, Splunk, AWS CloudWatch, or similar). Using AWS to provide cloud solutions for data platforms. Data pipelines, and orchestration tools. Working in More ❯

AWS S3, Azure Blob, MinIO, or similar) Proficiency in data parsing and transformation, handling structured and unstructured data Hands-on experience with ETL tools and data workflow orchestration (e.g., Apache Airflow, Luigi, Prefect) Strong programming skills in Python, SQL, or Scala Experience with open-source data processing tools (e.g., Kafka, Spark, Flink, Hadoop) Familiarity with database technologies (PostgreSQL More ❯

and reliability across our platform. Working format: full-time, remote. Schedule: Monday to Friday (the working day is 8+1 hours). Responsibilities: Design, develop, and maintain data pipelines using Apache Airflow . Create and support data storage systems (Data Lakes/Data Warehouses) based on AWS (S3, Redshift, Glue, Athena, etc.). Integrate data from various sources, including … pipeline into a standalone, scalable service. What we expect from you: 4+ years of experience as a Data Engineer, including 1+ year at a Senior level. Deep knowledge of Airflow : DAGs, custom operators, and monitoring. Strong command of PostgreSQL databases; familiarity with the AWS stack (S3, Glue or Redshift, Lambda, CloudWatch) is a significant plus. Excellent SQL skills and More ❯

Programming Languages: Python, Java, or Go. • Data Engineering Tools: Apache Kafka, Airflow (for orchestration), Spark (if needed for larger datasets). • OpenSearch/Elasticsearch: Indexing, querying, and optimizing. • Visualization Tools: Kibana, Grafana (for more advanced visualizations), React.js. • Cloud: AWS (ElasticSearch Service), or Azure (if using cloud infrastructure). Desired: • 15+ years of experience working in data engineering or More ❯

SQL and at least one programming language (e.g., Python) Familiarity with relational databases and data warehousing concepts Understanding of ETL concepts and tools Exposure to workflow orchestration tools like Apache Airflow, NiFi and Kafka Strong analytical and problem-solving skills Excellent communication and teamwork abilities Eagerness to learn and grow in a fast-paced environment Experience in Jupyter More ❯

to manage competing technical requirements across complex systems. Strong communication and stakeholder engagement skills, enabling you to translate technical solutions into business value. Desirable experience with tools like Snowflake, Apache Airflow, and AWS certification-or demonstrable equivalent knowledge-will help you thrive from day one. You'll benefit from Our compensation package includes a competitive salary, company bonus More ❯

data engineering or a related field. Proficiency in programming languages such as Python, Spark, SQL. Strong experience with SQL databases. Expertise in data pipeline and workflow management tools (e.g., Apache Airflow, ADF). Experience with cloud platforms (Azure preferred) and related data services. There's no place quite like BFS and we're proud of that. And it More ❯

with MLOps practices and model deployment pipelines Proficient in cloud AI services (AWS SageMaker/Bedrock) Deep understanding of distributed systems and microservices architecture Expert in data pipeline platforms (Apache Kafka, Airflow, Spark) Proficient in both SQL (PostgreSQL, MySQL) and NoSQL (Elasticsearch, MongoDB) databases Strong containerization and orchestration skills (Docker, Kubernetes) Experience with infrastructure as code (Terraform, CloudFormation More ❯

DATA ENGINEER - DBT/AIRFLOW/DATABRICKS 4-MONTH CONTRACT £450-550 PER DAY OUTSIDE IR35 This is an exciting opportunity for a Data Engineer to join a leading media organisation working at the forefront of data innovation. You'll play a key role in designing and building the data infrastructure that supports cutting-edge machine learning and LLM … to accelerate delivery of critical pipelines and platform improvements. THE ROLE You'll join a skilled data team to lead the build and optimisation of scalable pipelines using DBT, Airflow, and Databricks. Working alongside data scientists and ML engineers, you'll support everything from raw ingestion to curated layers powering LLMs and advanced analytics.Your responsibilities will include: Building and … maintaining production-grade ETL/ELT workflows with DBT and Airflow Collaborating with AI/ML teams to support data readiness for experimentation and inference Writing clean, modular SQL and Python code for use in Databricks Contributing to architectural decisions around pipeline scalability and performance Supporting the integration of diverse data sources into the platform Ensuring data quality, observability More ❯

Lead Data Engineer £75,000-£85,000 ️ AWS, Python, SQL, Airflow Brighton, hybrid working Analyse customer behaviour using AI & ML We are partnered with a private equity backed company who provide an AI-powered, guided selling platform that helps businesses improve online sales and customer experience. They are looking for a Lead Data Engineer to lead a small team … experience in a Senior Data Engineering role. Comfortable owning and delivering technical projects end-to-end. Strong in Python, SQL, and cloud platforms (AWS or comparable). Experience with Airflow, Snowflake, Docker (or similar). Familiarity with coaching and mentoring more junior engineers, leading 1-1s and check ins. Wider tech stack : AWS, Python, Airflow, Fivetran, Snowflake … Enhanced parental leave and pay If you are interested in finding out more, please apply or contact me directly! Lead Data Engineer £75,000-£85,000 ️ AWS, Python, SQL, Airflow Brighton, hybrid working Analyse customer behaviour using AI & ML Burns Sheehan Ltd will consider applications based only on skills and ability and will not discriminate on any grounds. More ❯

system performance and functionality. Requirements: -Active Top Secret/SCI Eligibility Clearance. -Minimum of 8 years of experience in data engineering or related work. -Proficiency in Java, AWS, Python, Apache Spark, Linux, Git, Maven, and Docker. -Experience maintaining an Apache Hadoop Ecosystem using tools like HBase, MapReduce, and Spark. -Knowledge of ETL processes utilizing Linux shell scripting, Perl … Python, and Apache Airflow. -Experience with AWS services such as CloudWatch, CloudTrail, ELB, EMR, KMS, SQS, SNS, and Systems Manager. -Experience in supporting, maintaining, and migrating JavaFX applications to modern cloud-native solutions. -Strong decision-making skills and domain knowledge. -Bachelor's Degree in a related field OR an additional 4 years of relevant experience in lieu of a More ❯

pipelines to power next-gen data products in the commodities industry. Ensure data quality using the latest analytics and monitoring tools. Design and build robust pipelines with tools like Airflow and DBT. Create scalable infrastructure on Azure using technologies like Terraform. Write clean, high-quality, reusable code aligned with best practices. Drive innovation by bringing your own ideas-your … in a fast-paced startup or agile environment. Strong background in schema design and dimensional data modeling. Able to communicate data architecture clearly with internal stakeholders. Experience with Azure, Airflow, DBT, Kubernetes, GitHub. Bonus points for: open-source contributions, an active GitHub profile, and curiosity for the latest in tech. A natural problem-solver who loves making things work. More ❯

pipelines to power next-gen data products in the commodities industry. Ensure data quality using the latest analytics and monitoring tools. Design and build robust pipelines with tools like Airflow and DBT. Create scalable infrastructure on Azure using technologies like Terraform. Write clean, high-quality, reusable code aligned with best practices. Drive innovation by bringing your own ideas-your … in a fast-paced startup or agile environment. Strong background in schema design and dimensional data modeling. Able to communicate data architecture clearly with internal stakeholders. Experience with Azure, Airflow, DBT, Kubernetes, GitHub. Bonus points for: open-source contributions, an active GitHub profile, and curiosity for the latest in tech. A natural problem-solver who loves making things work. More ❯

within ambitious software businesses. Specifically, you can expect to be involved in the following: Designing and developing full-stack data pipelines and platforms using modern tools such as dbt, Airflow, and cloud infrastructure Cleansing, enriching and modelling data to generate commercial insights and power C-level dashboards Delivering scalable solutions that support internal use cases and extend directly to … sales) and building tools that serve business needs Background in startups or scale-ups with high adaptability and a hands-on approach Experience with modern data tools (e.g. dbt, Airflow, CI/CD) and at least one cloud platform (AWS, GCP, Azure) Strong communication skills and a track record of credibility in high-pressure or client-facing settings BENEFITS More ❯

build batch data pipelines , insightful dashboards , and data products that serve new gaming customers and internal teams. What You'll Do Design and build scalable, maintainable data pipelines using Airflow and Snowflake Create data models and workflows that power dashboards, reports, and analytical products Collaborate with analysts, product teams, and game developers to deliver valuable insights Ensure data quality … documentation, and reliability across our data products Help shape the future of how data supports player engagement and product development What We're Looking For Hands-on experience with Airflow , Snowflake , and cloud data platforms (AWS/GCP) Strong SQL and Python skills for building and managing data workflows A knack for turning raw data into usable, impactful insights More ❯

build batch data pipelines , insightful dashboards , and data products that serve new gaming customers and internal teams. What You'll Do Design and build scalable, maintainable data pipelines using Airflow and Snowflake Create data models and workflows that power dashboards, reports, and analytical products Collaborate with analysts, product teams, and game developers to deliver valuable insights Ensure data quality … documentation, and reliability across our data products Help shape the future of how data supports player engagement and product development What We're Looking For Hands-on experience with Airflow , Snowflake , and cloud data platforms (AWS/GCP) Strong SQL and Python skills for building and managing data workflows A knack for turning raw data into usable, impactful insights More ❯

Data Storage & Databases: SQL & NoSQL Databases: Experience with databases like PostgreSQL, MySQL, MongoDB, and Cassandra. Big Data Ecosystems: Hadoop, Spark, Hive, and HBase. Data Integration & ETL: Data Pipelining Tools: Apache NiFi, Apache Kafka, and Apache Flink. ETL Tools: AWS Glue, Azure Data Factory, Talend, and Apache Airflow. AI & Machine Learning: Frameworks: TensorFlow, PyTorch, Scikit-learn, Keras More ❯

Spark). Experienced with Elasticsearch and Cloud Search. Hands-on experience with cloud platforms such as AWS, Azure, or Google Cloud Platform. Experience with data pipeline orchestration tools (e.g., Airflow, Luigi) and workflow automation tools (e.g., Jenkins, GitLab CI/CD). Familiarity with containerization and orchestration technologies (e.g., Docker, Kubernetes) is a plus. Data pipeline management Proven experience More ❯

platforms such as Azure or AWS, leveraging tools like Databricks, Data Factory, and Synapse Analytics. Ensuring best practices in DevOps, version control, and data governance. Managing orchestration workflows using Airflow or similar tools. Supporting AI teams by preparing high-quality, structured data for machine learning applications. Communicating technical concepts to non-technical stakeholders and making data solutions accessible to … and Python. Experience in Azure with tools like Databricks, Data Factory, and Synapse Analytics. Knowledge of data modeling techniques. Familiarity with DevOps and version control best practices. Experience with Airflow or other orchestration tools is a plus. Expertise in Machine Learning, especially NLP, is a plus. Certifications such as Azure DP-203 are a plus. Soft Skills: Ability to More ❯