926 to 946 of 946 Apache Airflow Jobs

CMEK encryption, lifecycle rules Batch pipelines: Spark on Dataproc for TB-scale feature backfills, Iceberg compaction, and daily S3 GCS incremental sync Streaming pipelines: Apache Beam on Dataflow for sub-5-min CDC ingestion with exactly-once semantics and PII assertion gates Schema registry: Avro/Protobuf schema versioning … exactly-once, side inputs, autoscaling Schema registry experience: Protobuf/Avro compatibility rules, breaking-change migrations in production Orchestration at scale: Flyte, Kubeflow Pipelines, Airflow, or Prefect - operated in production, ideally benchmarked two Multi-tenant data architecture: per-tenant isolation as a hard requirement, not a post-hoc concern ...

CMEK encryption, lifecycle rules Batch pipelines: Spark on Dataproc for TB-scale feature backfills, Iceberg compaction, and daily S3 GCS incremental sync Streaming pipelines: Apache Beam on Dataflow for sub-5-min CDC ingestion with exactly-once semantics and PII assertion gates Schema registry: Avro/Protobuf schema versioning … exactly-once, side inputs, autoscaling Schema registry experience: Protobuf/Avro compatibility rules, breaking-change migrations in production Orchestration at scale: Flyte, Kubeflow Pipelines, Airflow, or Prefect - operated in production, ideally benchmarked two Multi-tenant data architecture: per-tenant isolation as a hard requirement, not a post-hoc concern ...

Familiarity with Machine Learning pipelines or integrating AI models into production engineering systems. - Experience with large-scale data processing systems and orchestration framework (e.g., Apache Spark, Airflow, Ray, Kafka). - Open-minded and collaborative team player with strong willingness to help others. The US yearly salary range ...

Familiarity with Machine Learning pipelines or integrating AI models into production engineering systems. - Experience with large-scale data processing systems and orchestration framework (e.g., Apache Spark, Airflow, Ray, Kafka). - Open-minded and collaborative team player with strong willingness to help others. The US yearly salary range ...

migrate data sets into their AWS Snowflake platform. Designing and implementing dbt models that power clean, consistent metrics. Buidl out orchestartion pipelines using Airflow Creating clear documentation for dashboards, metrics, and models. Ensuring optimal performance and cost efficiency across Snowflake Supporting a stakeholder-first approach to data usability … access. KEY SKILLS AND REQUIREMENTS Extensive experience operating within Snowflake environments Airflow for data orchestration. Proficiency in SQL and dbt, with a strong understanding of data modeling best practices. Strong communication skills with the ability to translate complex needs into elegant data solutions. Focus on usability, documentation, and data ...

within the new architecture. You'll work across: Migrating legacy pipelines and workloads into Snowflake Building and refactoring pipelines using Python Developing orchestration through Airflow/MWAA Infrastructure and deployment using Terraform and Docker Supporting transformation and modelling using dbt Working across both legacy and modern AWS data environments … looking for a broad, hands-on Data Engineer who is comfortable working across data engineering, software engineering and infrastructure. Core experience: Python Airflow/MWAA Snowflake Terraform AWS Desirable experience: dbt Docker Redshift R Data platform migrations or modernisation programmes The ideal candidate will have worked within complex ...

high-performance application interfaces. You will ensure the data layer supports both rapid transactional workflows and deep analytical exploration. Data Orchestration: Utilize dbt and Airflow as your toolkit to clean, transform, and move data into your application-ready schemas, ensuring end-to-end data integrity. Modern Observability: Use Datadog … Centric Design: You should have a clear philosophy on how to make complex data-entry and data-analysis workflows intuitive. Workflow Orchestration: Experience with Airflow or similar tools to manage application-specific data transformations. Hands-on experience with Infrastructure as Code (Terraform), CI/CD and container orchestration (Kubernetes ...

Company DescriptionWise is a global technology company, building the best way to move and manage the world's money. Min fees. Max ease. Full speed. Whether people and businesses are sending money to another country ...

value, and/or FICC options trading Experience with Linux‐based, concurrent, high‐throughput, low‐latency software systems Experience with pipeline orchestration frameworks (e.g. Airflow, Dagster) Experience with streaming platforms (e.g. Kafka), data lake platforms (e.g. Delta Lake, Apache Iceberg), and relational databases Have a Bachelors or advanced ...

backend services using microservices or service mesh architectures Workflow Design: Build and manage task scheduling and long-running operations using tools like Temporal or Airflow Performance Optimization: Optimize LLM orchestration for performance and cost through caching, batching, and token monitoring Integrations & Data Services Data Pipeline Ownership: Implement and maintain … Data: Solid experience with AWS, Terraform, and a mix of relational (PostgreSQL) and Vector databases Orchestration: Practical experience with workflow engines like Temporal or Airflow and messaging systems like Kafka Preferred Qualifications: Experience with AI-driven planning systems or dialog state tracking Background in enterprise security (RBAC, SSO, OAuth2 ...

responsible for availability, latency, performance, efficiency, change management, monitoring, automation, emergency response, and capacity planning for the centralized platforms of Control-M, Automic, Airflow as well as Google services Cloud Composer, Workflows and Scheduler which are utilized across the Bank for workload automation/job scheduling/batch … days’ volunteering leave per year Your Key Responsibilities Develop solutions using Python, Kubernetes, Cloud services and API integrations Build and orchestrate REST APIs across Airflow, Google Cloud Platform, Control-M and Automic Drive the adoption of Infrastructure as Code (IaC) to automate provisioning, configuration, and management of infrastructure Deliver ...

United Kingdom to design, build, and maintain data pipelines on a modern AWS-native stack. You will use Spark/PySpark for distributed processing, Airflow for orchestration, and Terraform for infrastructure as code in a hybrid work arrangement with weekly travel to Newcastle office. You will contribute to data ...

Exploration of new data sources to test whether they can be of value to Zempler Bank Implement models into production using tools such as Airflow, PyFlink, BentoML, Git, Jenkins and Octopus Participate in business projects Manage the relationship with the model stakeholder Assist in requirements gathering and project scope … highly imbalanced datasets for binary classification problems Event streaming data engineering tools such as Flink or Spark Experience with MLOps and tools such as Airflow, BentoML and MLFlow Use of Docker, CI/CD and SDLC (e.g. JIRA, Git, Jenkins, Octopus) Why Zempler Bank What makes us Zempler ...

design, API development, and data modelling for production machine learning systems. Build and operate production solutions using technologies such as Python, TensorFlow, PyTorch, Redis, Airflow, Docker, Kubernetes, and cloud-native infrastructure. Design and deploy scalable batch and real-time ML pipelines supporting customer-facing applications. Develop and operationalise … designing, building, or scaling RAG systems. Experience in search, ranking, recommendations, personalisation, content intelligence, moderation, NLP, computer vision, or related AI domains. Experience with Airflow or comparable workflow orchestration platforms. Experience with cloud-native environments and container technologies, including Docker and Kubernetes. Experience developing model evaluation frameworks ...

foundational Large Language Models (LLMs), along with sub-agents, tools, and MCP integrations Architect end-to-end ML pipelines, integrating data processing (e.g. Spark, Airflow) with model training, evaluation, and deployment workflows Drive experimentation frameworks, including A/B testing and offline evaluation, to continuously improve model performance … fine‐tuning Large Language Models (LLMs) and building production AI Agents Proven experience designing scalable data and ML pipelines using tools such as Spark, Airflow, or similar distributed systems Demonstrated ability to operate as a senior individual contributor, influencing technical strategy and decision‐making without direct authority Experience partnering ...

GlobalLogic is seeking a Consultant Data Engineer to drive a major data modernization effort in the UK. You will build robust pipelines to move from on-prem Teradata to an AWS Lakehouse, converting hundreds of ...

experienced Full Stack Data Engineer to join our Customer Data Science team. You will design, build and maintain robust data pipelines, orchestration with Airflow, and Databricks-based solutions to empower data scientists with reliable data foundations and scalable platforms. You will collaborate across product and delivery partners, contribute ...

seeking a Python-focused data engineering specialist in the UK to design scalable data pipelines and reusable code. You'll leverage Airflow, Pandas, and SQLAlchemy, with hands-on GCP experience (BigQuery, Cloud Storage) and GitHub workflows to ensure reliable, production-grade deployments. Attention to debugging, scheduling, and observability with ...

Job Description Job Description About the company Our client is building an AI platform for construction capital program owners, helping teams collect, organize, and analyze project data and documents. Backed by $7.7M in seed funding ...

Job Description Job Description About the company Our client is building an AI platform for construction capital program owners, helping teams collect, organize, and analyze project data and documents. Backed by $7.7M in seed funding ...

Job Description Job Description Description: On-site 4/1 in Charlotte, NC Our client seeks a Senior AI Engineer to lead IVR and conversational AI initiatives for large-scale customer interactions in telecommunications and ...