51 to 75 of 141 Apache Airflow Jobs

environment Excellent communication and interpersonal skills Nice to Have: Experience with ML platforms such as SageMaker or similar tools Familiarity with workflow orchestration tools (Airflow, Kubeflow) Exposure to monitoring tools (CloudWatch, Prometheus, Grafana) Experience with NoSQL databases or time-series data systems Exposure to JavaScript/TypeScript or backend ...

environment Excellent communication and interpersonal skills Nice to Have: Experience with ML platforms such as SageMaker or similar tools Familiarity with workflow orchestration tools (Airflow, Kubeflow) Exposure to monitoring tools (CloudWatch, Prometheus, Grafana) Experience with NoSQL databases or time-series data systems Exposure to JavaScript/TypeScript or backend ...

checks across migration and ingestion pipelines. Optimize SQL queries and data processing performance (indexing, partitioning, tuning). Use orchestration and automation tools such as Airflow and Terraform. Apply data security and governance standards (GDPR, ISO 27001, NIST, PCI-DSS) using AWS features like IAM and KMS. Collaborate with stakeholders ...

/T-SQL with proven experience refactoring and optimising complex logic Hands-on experience with cloud data warehouses (e.g. BigQuery, Snowflake), ingestion tools (Airflow, Airbyte), and analytics tooling such as dbt and Power BI Strong working knowledge of Git/version control, CI/CD pipelines, and Python ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

with Python-based development workflows and tooling Technology Stack: Azure Cloud Services C# (.NET 10) Python React.js and modern JavaScript (ES6+) Snowflake (Data Warehouse) Apache Airflow Terraform dbt SQL Server ...

with Python-based development workflows and tooling Technology Stack: Azure Cloud Services C# (.NET 10) Python React.js and modern JavaScript (ES6+) Snowflake (Data Warehouse) Apache Airflow Terraform dbt SQL Server ...

have experience of building APIs, FastAPI preferred You're familiar with SQL, database management, Kubernetes, Docker, cloud platforms (Azure preferred), orchestration tools such as Airflow or Jenkins You're collaborative and pragmatic with great communication skills What's in it for you: Salary to £100k Discretionary bonus Impactful role ...

modern data stacks (e.g. Databricks, BigQuery, Snowflake) Strong understanding of data modelling for analytics and customer use cases Experience with workflow orchestration tools (e.g. Airflow, Step Functions) Experience leveraging AI tools or automation to improve engineering productivity Strong client-facing and stakeholder management skills Deep understanding of data modelling ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

Experience working within cloud environments (AWS or Azure) Strong SQL skills and experience working with large-scale datasets Familiarity with orchestration tools such as Airflow or similar Good understanding of data modelling and distributed data processing concepts Comfortable working across multiple client environments and stakeholders 🎁 What’s on Offer ...

include: Building data pipelines and data-intensive applications Developing in Python and SQL Working with distributed systems (e.g. Spark, Databricks) Managing data orchestration (e.g. Airflow) Ensuring data quality, reliability, and performance Data Engineer Requirements Strong data engineering experience Familiarity with cloud platforms (AWS, Azure or GCP) Experience with modern ...

Snowflake best practices for data ingestion and transformation. Enhancing performance and query optimisation, ensuring cost-effective and high-performing workloads. Working with dbt and Airflow to transform raw data into structured datasets for analytical consumption. Creating clear documentation to communicate technical processes to non-technical teams. KEY SKILLS ...

notebooks for analysis dbt and Databricks data platform with Delta Lake Lightdash as our BI tool and semantic layer Streamlit for interactive data applications Airflow for orchestration Kubernetes for application deployment CI/CD with GitHub and CircleCI Infrastructure on AWS, deployed with Terraform/Spacelift Why else ...

expand their Devops team (multiple roles). Key requirements: Strong experience with AWS services and infrastructure as code using Terraform Data engineering expertise using Apache Spark and Python, including development in Jupyter Notebooks Workflow orchestration and containerisation experience with Apache Airflow and Docker Version control ...

prompt engineering skills and the ability to critically evaluate AI-generated code for correctness, security, and financial accuracy Desirable, but no essential: Experience with Apache Airflow or similar workflow schedulers REST or GraphQL API design experience Knowledge of fixed income, FX, or derivatives products and associated ...

prompt engineering skills and the ability to critically evaluate AI-generated code for correctness, security, and financial accuracy Desirable, but no essential: Experience with Apache Airflow or similar workflow schedulers REST or GraphQL API design experience Knowledge of fixed income, FX, or derivatives products and associated ...

Experience with CI/CD, microservices, TDD/BDD. Knowledge of secure system design. Desirable (AI & Data) Vector DBs: Milvus, Weaviate, Pinecone Data orchestration: Airflow, Prefect, Dagster Search: ElasticSearch/OpenSearch AI frameworks: LangChain, Hugging Face MLOps: MLflow, Kubeflow If you are looking for an opportunity to work ...

responsibilities Own, assess, improve, or rebuild existing ETL/ELT pipelines Build reliable data workflows using Python Work with orchestration tools such as Airflow or similar Improve data quality, validation, lineage, and troubleshooting Collaborate with consultants and non-technical stakeholders Help translate messy business/client data into structured ...

data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform or CDK) Experience with workflow orchestration tools such as Airflow or similar Ability to design scalable architectures, make pragmatic trade-offs, and communicate effectively with both technical and non-technical stakeholders Processing Your Data ...

projects are approached and ensuring delivery standards are upheld Designing and governing scalable pipelines using BigQuery, Dataflow, Datastream, Data Fusion, Dataproc and Cloud Composer (Airflow) Leading best practice across dbt, Terraform, Docker and CI/CD Driving Data Mesh adoption, domain ownership, and interoperable data products Setting modelling standards ...

familiarity with common AI/ML libraries (scikit-learn, PyTorch, HuggingFace, OpenAI API). Bonus Points Experience with data engineering tools (SQL, pandas, dbt, Airflow). Exposure to LLMs or generative AI in a practical context. GitHub profile demonstrating collaborative projects and consistent contributions. Who we are: Here ...

platforms and pipelines (batch and real-time ingestion). Hands-on familiarity with: Snowflake (or equivalent cloud data warehouse), Looker (or other BI tools), Airflow, Pub/Sub, or similar orchestration and ingestion frameworks. Experience with data infrastructure monitoring and optimization. Strong communication skills. Ability to translate business goals ...

data stack including a mix of structured and unstructured data sources, orchestration tooling, and distributed storage. RDS Postgres, MongoDB, AWS Athena, Parquet, AWS Glue, Airflow, Python, Docker, S3, GraphQL, REST. You won't know all of it, you'll be strong in your core area and curious about ...

maintain ETL pipelines and data workflows Build automated processes for data ingestion, transformation, and validation Implement scheduling/orchestration tools (cron, AWS services, Airflow, or equivalent) Ensure processes run without manual intervention Scripting & Process Automation Develop scripts using Python, Bash, or PowerShell Automate data pulls, file processing, and pipeline ...