151 to 175 of 196 Apache Airflow Jobs

for optimizing ML inference performance. Hands-on expertise designing and managing batch and real-time ML pipelines/workflows for structured and unstructured data on AWS (SageMaker, ECS, EKS, Airflow). Familiarity with LLMOps and observability tools (Langsmith, Opik, Ragas, Weights & Biases) for monitoring, evaluation, and governance. Excellent communication skills and the ability to collaborate effectively across teams. More ❯

in delta one, store of value, and/or FICC options trading Experience with Linux-based, concurrent, high-throughput, low-latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced degree in Computer Science, Mathematics More ❯

data pipelines Strong understanding of data modeling, schema design, and lakehouse principles Familiarity with data governance, lineage, and quality frameworks Experience working on enterprise-class applications Preferred: Experience with Apache Spark Hands-on experience with AWS data services (e.g., S3, Glue, Lambda, MSK) Capable in batch or streaming data processing using technologies such as Spark, Kafka, Flink, and DLT … Proficient in CI/CD pipelines, automated testing, code quality enforcement, and environment management for production-grade data systems Expert in orchestration and transformation frameworks such as Airflow, dbt, and Dagster, along with cloud-native platforms like Databricks Financial services or FinTech industry experience Knowledge and Skills: Operates with full autonomy on large-scale, complex data projects. Go to More ❯

analytics, and software development, contributing to innovative products and services that drive real business value. Key Responsibilities Design and implement robust data pipelines using tools such as dbt and Airflow . Collaborate with clients to understand business challenges and deliver tailored data solutions. Transform raw data into structured, actionable datasets for use in AI , predictive modelling , and platform development … the ability to engage stakeholders and translate technical work into business outcomes. Hands-on experience with cloud platforms (preferably AWS or Snowflake) and ETL/ELT orchestration tools (e.g. Airflow, dbt). Desirable Skills Familiarity with tools such as MongoDB , Glue , Athena , and React . Experience in API development and working with modern data infrastructure. Ability to work independently More ❯

analytics, and software development, contributing to innovative products and services that drive real business value. Key Responsibilities Design and implement robust data pipelines using tools such as dbt and Airflow . Collaborate with clients to understand business challenges and deliver tailored data solutions. Transform raw data into structured, actionable datasets for use in AI , predictive modelling , and platform development … the ability to engage stakeholders and translate technical work into business outcomes. Hands-on experience with cloud platforms (preferably AWS or Snowflake) and ETL/ELT orchestration tools (e.g. Airflow, dbt). Desirable Skills Familiarity with tools such as MongoDB , Glue , Athena , and React . Experience in API development and working with modern data infrastructure. Ability to work independently More ❯

we are relied upon to ensure our systems are trusted, reliable and available.The technology underpinning these capabilities includes industry leading data and analytics products such as Snowflake, Astronomer/Airflow, Kubernetes , DBT, Tableau, Sisense , Collibra, and Kafka/Debezium . Our mission is to enable frictionless experiences for our AIS colleagues and customers so that they can openly and … governance by identifying capability gaps, implementing necessary tooling and processes, and promoting DataOps through leadership and user feedback initiatives. Requirements: Deploy and govern modern data stack technologies (e.g., Snowflake, Airflow, DBT, Fivetran, Airbyte, Tableau, Sisense, AWS, GitHub, Terraform, Docker) at enterprise scale for data engineering workloads. Develop deployable, reusable ETL/ELT solutions using Python, advanced SQL, and Jinja More ❯

Programming Languages: Python, Java, or Go. • Data Engineering Tools: Apache Kafka, Airflow (for orchestration), Spark (if needed for larger datasets). • OpenSearch/Elasticsearch: Indexing, querying, and optimizing. • Visualization Tools: Kibana, Grafana (for more advanced visualizations), React.js. • Cloud: AWS (ElasticSearch Service), or Azure (if using cloud infrastructure). Desired: • 15+ years of experience working in data engineering or More ❯

through data science projects Awareness of data security best practices Experience in agile environments You would benefit from having: Understanding of data storage and processing design choices Familiarity with Apache Spark or Airflow Experience with parallel computing Candidates should be able to reliably commute or plan to relocate to Coventry before starting work. The role requires a Data More ❯

ships, aircraft). Required Experience: Active Secret clearance. 4-7 years in data engineering, preferably within secure or classified environments. Proficiency in Python, Spark, SQL, and orchestration tools like Apache Airflow. Hands-on experience with data serialization formats such as protobuf, Arrow, FlatBuffers, or Cap'n Proto. Familiarity with data storage formats like Parquet or Avro. Experience with modern … analytic storage technologies such as Apache Iceberg or DuckDB. Binary message parsing experience. Strong understanding of classified data handling, secure networking, and compliance in high-side or air-gapped environments. Preferred Experience: Familiarity with IC standards (UDS, IC ITE) and secure cloud environments (e.g., AWS GovCloud, C2S). Experience deploying LLMs or machine learning models within classified network environments More ❯

in Python, ensuring scalability and reliability. Extract data from multiple external sources via APIs, and where necessary, web scraping/browser automation (Playwright, Selenium, Puppeteer). Orchestrate pipelines using Airflow, and manage data quality workflows. Model and transform data in SQL and Snowflake to create clean, analytics-ready datasets. Ensure data quality, observability, and governance across workflows. Collaborate closely … who bring: Strong hands-on experience with Python for API ingestion, pipeline automation, and data transformation. Solid SQL skills with Snowflake (or similar cloud data warehouses). Experience with Airflow or other orchestration tools. Knowledge of data modelling, warehouse performance optimisation, and governance. Cloud experience (AWS preferred; Terraform/Docker a plus). Nice-to-have: browser automation/ More ❯

Responsibilities: Design, build, and maintain scalable and reliable ETL/ELT data pipelines using Python, PySpark, and SQL. Develop and manage data workflows and orchestration using tools such as Airflow or similar. Optimize data processes for performance, scalability, and cost-efficiency, particularly in cloud environments. Collaborate with data scientists, analysts, and product teams to understand data needs and deliver … is a plus. Experience building and managing data lakes and data warehouses. Strong understanding of distributed systems and big data processing. Experience with data workflow orchestration tools such as Airflow, dbt , or similar. Solid understanding of data modeling, data quality, and data governance concepts. Familiarity with CI/CD pipelines and infrastructure as code (e.g., Terraform or CloudFormation) is More ❯

Responsibilities: Design, build, and maintain scalable and reliable ETL/ELT data pipelines using Python, PySpark, and SQL. Develop and manage data workflows and orchestration using tools such as Airflow or similar. Optimize data processes for performance, scalability, and cost-efficiency, particularly in cloud environments. Collaborate with data scientists, analysts, and product teams to understand data needs and deliver … is a plus. Experience building and managing data lakes and data warehouses. Strong understanding of distributed systems and big data processing. Experience with data workflow orchestration tools such as Airflow, dbt , or similar. Solid understanding of data modeling, data quality, and data governance concepts. Familiarity with CI/CD pipelines and infrastructure as code (e.g., Terraform or CloudFormation) is More ❯

SonarQube, Cypress, PowerShell, C#, and Databricks Experience with Docker, SQL, Angular, Spring Boot, Nifi, AWS, python, scala, shell scripting, and XML processing Experience in AWS solution architecture Maintaining the Apache Hadoop Ecosystem, especially utilizing HBase, MapReduce, and Spark. ETL processes utilizing Linux shell scripting, Perl, Python, and Apache Airflow. AWS services such as CloudWatch, CloudTrail, ELB, EMR, KMS … modeling, and advanced analytics Databricks and Lakehouse architectures AWS OpenSearch Experience with AWS, Splunk, Databricks, and other Oracle/SQL based platforms. Experience with python, Microsoft VBA, and Databricks Apache Niagra Files (NiFi) Apache TIKA Databricks and Lakehouse architecture ElasticSearch AWS SQS Informatica and custom software components ElasticSearch and OpenSearch .NET, C#, Javascript, and Java, Python Terraform Experience More ❯

Title: GCP -Data Engineer. Location: Philadelphia PA (Can submit who are willing to relocate) GCP Data Engineer - GCP Dataflow and Apache Beam (Key skills) Primary Skills- PySpark, Spark, Python, Big Data, GCP, Apache Beam, Dataflow, Airflow, Kafka and BigQuery GFO, Google Analytics Javascript is Must Strong Experience with Dataflow and BigQuery A person should have leading the … Platforms (preferably GCP) provided Big Data technologies Hands-on experience with real-time streaming processing as well as high volume batch processing, and skilled in Advanced SQL, GCP BigQuery, Apache Kafka, Data-Lakes, etc. Hands-on Experience in Big Data technologies - Hadoop, Hive, and Spark, and an enterprise-scale Customer Data Platform (CDP) Experience in at least one programming …/PySpark, columnar datastores (BigQuery preferred), DevOps processes/tooling (CI/CD, GitHub Actions), infrastructure as code frameworks (Terraform), BI Tools (e.g. DOMO, Tableau, Looker,), pipeline orchestration (eg. Airflow) Fluency in data science/machine learning basics (model types, data prep, training process, etc.) Experience using version control systems (Git is strongly preferred) Experience with data governance and More ❯

development lifecycles, cloud technologies and modern engineering practices. ● Experience with the following technologies: o Cloud Provider: AWS o Languages: Python, PHP, Rust & SQL o Hosting: Kubernetes o Tooling & Analytics: Airflow, RabbitMQ, Apache Spark, PowerBI ● Proven ability to complete projects according to outlined scope, budget, and timeline ● Experience with industry standard tools such as Microsoft products, Jira, confluence, project More ❯

development lifecycles, cloud technologies and modern engineering practices. ● Experience with the following technologies: o Cloud Provider: AWS o Languages: Python, PHP, Rust & SQL o Hosting: Kubernetes o Tooling & Analytics: Airflow, RabbitMQ, Apache Spark, PowerBI ● Proven ability to complete projects according to outlined scope, budget, and timeline ● Experience with industry standard tools such as Microsoft products, Jira, confluence, project More ❯

from an accredited college in a related discipline YEARS OF EXPERIENCE: 8 Years Minimum SKILLS/CERTIFICATIONS: Experience with Atlassian Software products (JIRA, Confluence, Service Desk, etc.) Maintaining the Apache Hadoop Ecosystem, especially utilizing HBase, MapReduce, and Spark. ETL processes utilizing Linux shell scripting, Perl, Python, and Apache Airflow. AWS services such as CloudWatch, CloudTrail, ELB, EMR, KMS … SQS, SNS, and Systems Manager. Vue.js, ASP.NET (C#), NodejsReact, JavaScript, HTML, CSS, PostgreSQL, Liquibase, Elasticsearch, and Git. Ansible Apache Niagra Files (NiFi) Apache TIKA Databricks and Lakehouse architecture ElasticSearch, Splunk AWS SQS Ability to deliver an advanced visual analytic application to include developing data analytics for desktop and web-developed visual analytic software; facilitating the bulk analysis of More ❯

automation, reliability, and agility. Key Responsibilities Design, build, and optimise data pipelines across a modern data platform. Ingest, clean, and transform data using tools such as dbt, Snowflake, and Airflow . Collaborate with cross-functional teams to deliver data products aligned to business priorities. Develop scalable data models that support BI and analytics platforms including Tableau and Power BI. … and optimise complex queries. Hands-on experience with dbt (including testing and layered modelling). Practical knowledge of Snowflake for loading, transforming, and exporting datasets. Experience building and managing Airflow DAGs for pipeline orchestration. Understanding of BI tool requirements (e.g., Tableau, Power BI) and related performance considerations. Advanced Excel capability, including pivot tables and complex formulas. Familiarity with data More ❯

you'll be doing at BRITA You will play an active role in setting up and operating our new Data Analytics Platform, using modern technologies such as Databricks, dbt, Airflow, Terraform and Azure Data Services. At the same time, you will manage our existing BI landscape and help shape the transition to the new architecture. Your responsibilities will include … from us A hybrid data landscape with proven on-premise technologies (SQL Server, SSIS, SSAS, Power BI) and a modern Data Analytics Platform with technologies such as Databricks, dbt, Airflow, Azure Data Services, Terraform, GitHub, Python, and Fabric currently under development. A motivated team with principles such as trust, personal responsibility, transparency and continuous development. A structured operating process More ❯

automated pipelines, and shaping the foundational framework for how we leverage data to succeed. What You'll Do You'll develop and maintain data pipelines and automated processes in Airflow and Python You'll create SQL data models with dbt to power dashboards and applications You'll integrate third-party APIs and databases into our data flows You'll … notebook analytics and collaboration Circle CI for continuous deployment AWS cloud infrastructure Kubernetes for data services and task orchestration Google Analytics, Amplitude and Firebase for client applications event processing Airflow for job scheduling and tracking Parquet and Delta file formats on S3 for data lake storage Streamlit for data applications Why else you'll love it here Wondering what More ❯

term Interview Criteria: Telephonic + Zoom Direct Client Requirement Role: AWS Data Engineer We are seeking a skilled AWS Data Engineer who has experience working with Python, PySpark, lambda, Airflow, and Snowflake.Responsibilities: Design, build, and optimize ETLs using Python, PySpark, lambda, Airflow and other AWS services. Create SQL queries to segment, manipulate, and formatdata. Build automations to ingest … ingest data into Amazon Redshift for analytics and reporting. Requirements: Minimum 5 years of experience as Data Engineer. 3+ years of Python, PySpark, and Lambda. Must have experience with Airflow and Snowflake. Advanced SQL query development proficiency Understanding of data modelling principles and techniques. Knowledge of data security best practices and compliance requirements. Note: If you are interested, please More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯

colleagues. Nice to Have Skills Experience working with GCP (BigQuery) or other modern cloud-native data warehouses (e.g. Snowflake, Redshift). Familiarity with data pipelining and orchestration systems (e.g. Airflow). Understanding of modern analytics architectures and data visualisation tools (we use Preset.io/Apache Superset). Exposure to CI/CD pipelines (GitLab CI preferred). Experience More ❯