151 to 175 of 185 Permanent Apache Airflow Jobs

SonarQube, Cypress, PowerShell, C#, and Databricks Experience with Docker, SQL, Angular, Spring Boot, Nifi, AWS, python, scala, shell scripting, and XML processing Experience in AWS solution architecture Maintaining the Apache Hadoop Ecosystem, especially utilizing HBase, MapReduce, and Spark. ETL processes utilizing Linux shell scripting, Perl, Python, and Apache Airflow. AWS services such as CloudWatch, CloudTrail, ELB, EMR, KMS … modeling, and advanced analytics Databricks and Lakehouse architectures AWS OpenSearch Experience with AWS, Splunk, Databricks, and other Oracle/SQL based platforms. Experience with python, Microsoft VBA, and Databricks Apache Niagra Files (NiFi) Apache TIKA Databricks and Lakehouse architecture ElasticSearch AWS SQS Informatica and custom software components ElasticSearch and OpenSearch .NET, C#, Javascript, and Java, Python Terraform Experience More ❯

Title: GCP -Data Engineer. Location: Philadelphia PA (Can submit who are willing to relocate) GCP Data Engineer - GCP Dataflow and Apache Beam (Key skills) Primary Skills- PySpark, Spark, Python, Big Data, GCP, Apache Beam, Dataflow, Airflow, Kafka and BigQuery GFO, Google Analytics Javascript is Must Strong Experience with Dataflow and BigQuery A person should have leading the … Platforms (preferably GCP) provided Big Data technologies Hands-on experience with real-time streaming processing as well as high volume batch processing, and skilled in Advanced SQL, GCP BigQuery, Apache Kafka, Data-Lakes, etc. Hands-on Experience in Big Data technologies - Hadoop, Hive, and Spark, and an enterprise-scale Customer Data Platform (CDP) Experience in at least one programming …/PySpark, columnar datastores (BigQuery preferred), DevOps processes/tooling (CI/CD, GitHub Actions), infrastructure as code frameworks (Terraform), BI Tools (e.g. DOMO, Tableau, Looker,), pipeline orchestration (eg. Airflow) Fluency in data science/machine learning basics (model types, data prep, training process, etc.) Experience using version control systems (Git is strongly preferred) Experience with data governance and More ❯

development lifecycles, cloud technologies and modern engineering practices. ● Experience with the following technologies: o Cloud Provider: AWS o Languages: Python, PHP, Rust & SQL o Hosting: Kubernetes o Tooling & Analytics: Airflow, RabbitMQ, Apache Spark, PowerBI ● Proven ability to complete projects according to outlined scope, budget, and timeline ● Experience with industry standard tools such as Microsoft products, Jira, confluence, project More ❯

development lifecycles, cloud technologies and modern engineering practices. ● Experience with the following technologies: o Cloud Provider: AWS o Languages: Python, PHP, Rust & SQL o Hosting: Kubernetes o Tooling & Analytics: Airflow, RabbitMQ, Apache Spark, PowerBI ● Proven ability to complete projects according to outlined scope, budget, and timeline ● Experience with industry standard tools such as Microsoft products, Jira, confluence, project More ❯

from an accredited college in a related discipline YEARS OF EXPERIENCE: 8 Years Minimum SKILLS/CERTIFICATIONS: Experience with Atlassian Software products (JIRA, Confluence, Service Desk, etc.) Maintaining the Apache Hadoop Ecosystem, especially utilizing HBase, MapReduce, and Spark. ETL processes utilizing Linux shell scripting, Perl, Python, and Apache Airflow. AWS services such as CloudWatch, CloudTrail, ELB, EMR, KMS … SQS, SNS, and Systems Manager. Vue.js, ASP.NET (C#), NodejsReact, JavaScript, HTML, CSS, PostgreSQL, Liquibase, Elasticsearch, and Git. Ansible Apache Niagra Files (NiFi) Apache TIKA Databricks and Lakehouse architecture ElasticSearch, Splunk AWS SQS Ability to deliver an advanced visual analytic application to include developing data analytics for desktop and web-developed visual analytic software; facilitating the bulk analysis of More ❯

Python, Java, Ansi SQL, Abinitio, Spark Experience in building data integration solutions using the AB initio ETL (link removed) Creating efficient and scalable data processing pipelines and applications using Apache spark Proficient in understanding distributed computing principles. Creating ,optimizing, and maintaining directed acyclic graphs(DAGs) in Python to define and orchestrate data pipelines and automated tasks. Implementing , scheduling, and … monitoring complex data workflows, ensuring timely and accurate data processing. Identifying, diagnosing, and resolving issues within airflow workflows, and optimizing DAGs for performance, scalability, and resource efficiency. Good understanding of cloud technology. Must have strong technical experience in Desing Mapping Specifications HLD LLD. Must have the ability to relate to both business and technical members of the team and … integration pipelines, ensuring adaptability to evolving business needs. Building data integration solutions using the Ab initio ETL (link removed) Creating efficient and scalable data processing pipelines and applications using Apache spark. Implementing, scheduling, and monitoring complex data workflows, ensuring timely and accurate data processing. Implement and manage agents for monitoring, logging, and automation within aws environments Handling migration from More ❯

automation, reliability, and agility. Key Responsibilities Design, build, and optimise data pipelines across a modern data platform. Ingest, clean, and transform data using tools such as dbt, Snowflake, and Airflow . Collaborate with cross-functional teams to deliver data products aligned to business priorities. Develop scalable data models that support BI and analytics platforms including Tableau and Power BI. … and optimise complex queries. Hands-on experience with dbt (including testing and layered modelling). Practical knowledge of Snowflake for loading, transforming, and exporting datasets. Experience building and managing Airflow DAGs for pipeline orchestration. Understanding of BI tool requirements (e.g., Tableau, Power BI) and related performance considerations. Advanced Excel capability, including pivot tables and complex formulas. Familiarity with data More ❯

you'll be doing at BRITA You will play an active role in setting up and operating our new Data Analytics Platform, using modern technologies such as Databricks, dbt, Airflow, Terraform and Azure Data Services. At the same time, you will manage our existing BI landscape and help shape the transition to the new architecture. Your responsibilities will include … from us A hybrid data landscape with proven on-premise technologies (SQL Server, SSIS, SSAS, Power BI) and a modern Data Analytics Platform with technologies such as Databricks, dbt, Airflow, Azure Data Services, Terraform, GitHub, Python, and Fabric currently under development. A motivated team with principles such as trust, personal responsibility, transparency and continuous development. A structured operating process More ❯

automated pipelines, and shaping the foundational framework for how we leverage data to succeed. What You'll Do You'll develop and maintain data pipelines and automated processes in Airflow and Python You'll create SQL data models with dbt to power dashboards and applications You'll integrate third-party APIs and databases into our data flows You'll … notebook analytics and collaboration Circle CI for continuous deployment AWS cloud infrastructure Kubernetes for data services and task orchestration Google Analytics, Amplitude and Firebase for client applications event processing Airflow for job scheduling and tracking Parquet and Delta file formats on S3 for data lake storage Streamlit for data applications Why else you'll love it here Wondering what More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

you'll be doing at BRITA You will play an active role in setting up and operating our new Data Analytics Platform, using modern technologies such as Databricks, dbt, Airflow, Terraform and Azure Data Services. At the same time, you will manage our existing BI landscape and help shape the transition to the new architecture. Your responsibilities will include … from us A hybrid data landscape with proven on-premise technologies (SQL Server, SSIS, SSAS, Power BI) and a modern Data Analytics Platform with technologies such as Databricks, dbt, Airflow, Azure Data Services, Terraform, GitHub, Python, and Fabric currently under development. A motivated team with principles such as trust, personal responsibility, transparency and continuous development. A structured operating process More ❯

you'll be doing at BRITA You will play an active role in setting up and operating our new Data Analytics Platform, using modern technologies such as Databricks, dbt, Airflow, Terraform and Azure Data Services. At the same time, you will manage our existing BI landscape and help shape the transition to the new architecture. Your responsibilities will include … from us A hybrid data landscape with proven on-premise technologies (SQL Server, SSIS, SSAS, Power BI) and a modern Data Analytics Platform with technologies such as Databricks, dbt, Airflow, Azure Data Services, Terraform, GitHub, Python, and Fabric currently under development. A motivated team with principles such as trust, personal responsibility, transparency and continuous development. A structured operating process More ❯

you'll be doing at BRITA You will play an active role in setting up and operating our new Data Analytics Platform, using modern technologies such as Databricks, dbt, Airflow, Terraform and Azure Data Services. At the same time, you will manage our existing BI landscape and help shape the transition to the new architecture. Your responsibilities will include … from us A hybrid data landscape with proven on-premise technologies (SQL Server, SSIS, SSAS, Power BI) and a modern Data Analytics Platform with technologies such as Databricks, dbt, Airflow, Azure Data Services, Terraform, GitHub, Python, and Fabric currently under development. A motivated team with principles such as trust, personal responsibility, transparency and continuous development. A structured operating process More ❯

and governance through robust access controls, including RBAC, SSO, token policies, and pseudonymisation frameworks. Develop resilient data flows for both batch and streaming workloads using technologies such as Kafka, Airflow, DBT, and Terraform. Shape data strategy and standards by contributing to architectural decisions, authoring ADRs, and participating in reviews, data councils, and platform enablement initiatives. Qualifications What we’d … requirements Direct exposure to cloud-native data infrastructures (Databricks, Snowflake) especially in AWS environments is a plus Experience in building and maintaining batch and streaming data pipelines using Kafka, Airflow, or Spark Familiarity with governance frameworks, access controls (RBAC), and implementation of pseudonymisation and retention policies Exposure to enabling GenAI and ML workloads by preparing model-ready and vector More ❯

Join our rapidly expanding team as a hands-on Cloud Data Analytics Platform Engineer and play a pivotal role in shaping the future of data at Citi. We're building a cutting-edge, multi-cloud data analytics platform that empowers More ❯

Azure/Microsoft Fabric/Data Factory) and modern data warehouse technologies (Databricks, Snowflake) Experience with database technologies such as RDBMS (SQL Server, Oracle) or NoSQL (MongoDB) Knowledge in Apache technologies such as Spark, Kafka and Airflow to build scalable and efficient data pipelines Ability to design, build, and deploy data solutions that explore, capture, transform, and utilize More ❯

points and decision engines (e.g., XACML, OPA). • Familiarity with secure data transfer protocols, cross-domain solutions (CDS), and data protection. • Proficient with data management tools and languages (e.g., Apache NiFi, Kafka, Airflow, Python, Spark). • Excellent problem-solving skills and the ability to collaborate with cross-functional teams. More ❯

Job Title: Data Engineer Salary: £40k - £60k + Excellent Benefits Package Location: Huddersfield (Hybrid) Key Skills: SQL, PowerBI, Airflow Summary A new role has opened up for a Data Engineering with SQL, BI, Cloud and Airflow DAG to join a media-focused business. The role has opened up as the company are heavily investing, and have exciting plans … responsibilities will cover: Develop data models to support company Business Intelligence Write and optimize complex SQL queries Build, maintain and improve data pipelines Transform data using DBT, Snowflake and Airflow Ensure data is handled correctly and to relevant standards Collaborate with tech teams to collectively solve shared data challenges Key Skills SQL DBT Snowflake Airflow DAG Cloud Platform … successful candidate to Oscar. Email: to recommend someone for this role Job Title: Data Engineer Salary: £40k - £60k + Excellent Benefits Package Location: Huddersfield (Hybrid) Key Skills: SQL, PowerBI, Airflow Oscar Associates (UK) Limited is acting as an Employment Agency in relation to this vacancy. To understand more about what we do with your data please review our privacy More ❯

core of VAULT is big data at scale. Our systems handle massive ingestion pipelines, long-term storage, and high-performance querying. We leverage distributed technologies (Kafka, Spark, Flink, Cassandra, Airflow, etc.) to deliver resilient, low-latency access to trillions of records, while continuously optimizing for scalability, efficiency, and reliability. We'll trust you to: Build high-performance, distributed data … pipelines and services Lead technical direction and ensure alignment with business needs Collaborate across teams using modern open-source tech (Kafka, FastAPI, Airflow, Spark, etc.) Mentor junior engineers and contribute to an inclusive team culture You will need to have: Bachelor's in Computer Science, Engineering, or related field (or equivalent experience) 4+ years of professional experience in an … Deep background in distributed, high-volume, high-availability systems Fluency in AI development tools We would love to see: Experience with big data ecosystems (Kafka, Spark, Flink, Cassandra, Redis, Airflow) Familiarity with cloud platforms (AWS, Azure, GCP) and S3-compatible storage SaaS/PaaS development experience Container technologies (Docker, Kubernetes) Salary Range = 160000 - 240000 USD Annually + Benefits + More ❯

Collaborate with ML researchers and biologists to translate raw data into actionable insights and high-quality training data Scale distributed systems using Kubernetes, Terraform, and orchestration tools such as Airflow, Flyte, or Temporal Write clean, extensible, and well-tested code to ensure long-term maintainability and collaboration across teams About You We are looking for data and platform engineers … Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform, orchestration frameworks such as Airflow/Flyte/Temporal) Hands-on cloud engineering experience (AWS, GCP, or Azure) Strong software engineering fundamentals, with a track record of writing maintainable, testable, and extensible code Familiarity More ❯

Collaborate with ML researchers and biologists to translate raw data into actionable insights and high-quality training data Scale distributed systems using Kubernetes, Terraform, and orchestration tools such as Airflow, Flyte, or Temporal Write clean, extensible, and well-tested code to ensure long-term maintainability and collaboration across teams About You We are looking for data and platform engineers … Experience designing and implementing large-scale data storage systems (feature stores, timeseries databases, warehouses, or object stores) Strong distributed systems and infrastructure skills (Kubernetes, Terraform, orchestration frameworks such as Airflow/Flyte/Temporal) Hands-on cloud engineering experience (AWS, GCP, or Azure) Strong software engineering fundamentals, with a track record of writing maintainable, testable, and extensible code Familiarity More ❯