76 to 100 of 221 Apache Airflow Jobs

locally-hosted solutions. Technical Requirements Languages Expert-level Python and advanced SQL . Data Engineering Experience with ETL/ELT and orchestration tools like Apache Airflow or NiFi . On-Prem Tech Proficiency with Hadoop/HDFS , Spark , and containerization via Docker/Kubernetes (K3s/OpenShift) . ...

data platforms and backend systems. Familiarity with data governance frameworks, privacy compliance, and automated data quality checks. Hands-on experience with big data tools (Airflow, Databricks) and data observability platforms. Collaborative mindset and experience working with cross-functional teams including ML and analytics specialists. Curiosity and enthusiasm for continuous ...

projects are approached and ensuring delivery standards are upheld Designing and governing scalable pipelines using BigQuery, Dataflow, Datastream, Data Fusion, Dataproc and Cloud Composer (Airflow) Leading best practice across dbt, Terraform, Docker and CI/CD Driving Data Mesh adoption, domain ownership, and interoperable data products Setting modelling standards ...

serving, scaling and monitoring workflows (e.g., BentoML, MLflow, Vertex AI, AWS Sagemaker) Experience building streaming + batch data ingestion and transformation pipelines (Spark/Airflow/Beam) Mobile and front-end web application development experience What We Offer At Newpage, we're building a company that works smart ...

serving, scaling and monitoring workflows (e.g., BentoML, MLflow, Vertex AI, AWS Sagemaker) Experience building streaming + batch data ingestion and transformation pipelines (Spark/Airflow/Beam) Mobile and front-end web application development experience What We Offer At Newpage, we’re building a company that works smart ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

development experience. Experience building lakehouse architectures using Delta Lake. Strong cloud experience in Azure (preferred), AWS, or GCP. Experience with orchestration tools (ADF, Airflow, dbt, or similar). Certifications – Databricks Certified Data Engineer/Azure Data Engineer What’s in it for you: Fully remote role with flexible working ...

Familiarity with cloud platforms such as AWS, Azure or Google Cloud and associated MLOps practices. Experience with automation and orchestration tooling such as UiPath, Airflow or Kubeflow. Experience and Capability Strong experience delivering AI or automation solutions within enterprise environments. Ability to work with cross functional teams and translate ...

Candidates will be asked to rate their knowledge of: Languages: Python, Java, Scala Compute/Distributed: RAY , Spark, Databricks, Hadoop Orchestration/Cloud: Kafka, Airflow, Prefect, AWS Storage/Warehouse: Clickhouse, Snowflake, Redshift, Greenplum Note: Experience with Petabyte/Exabyte scale is a strong signal for this role. Randstad ...

morale Strong communication and stakeholder management skills Highly Desirable Experience in cloud-native or containerised environments Exposure to data tooling or workflow orchestration (e.g., Airflow, dbt) Experience scaling teams or improving engineering processes Experience mentoring early- to mid-career engineers Why Consider This Role? Real ownership over both technical ...

Data Engineer - FinTech - Newcastle (Tech stack: Data Engineer, SQL, Python, AWS, Git, Airflow, Data Pipelines, Data Platforms, Programmer, Developer, Architect, Data Engineer) Our client is a trailblazer in the FinTech space, known for delivering innovative technology solutions to global financial markets. They are expanding their engineering capability in Newcastle ...

Scala experience Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs, predictive models ...

Scala experience Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs, predictive models ...

Azure, or Google Cloud) including data lakes, warehouses, and lakehouse solutions. Transform and process structured and unstructured data using modern ETL/ELT frameworks (Apache Spark, Airflow, dbt). Collaborate closely with product managers, analysts, and software developers to ensure seamless integration and high-quality data availability. Develop ...

staff to build long-term architectural capability. Preferred Tech Stack Expertise Google Cloud Platform (BigQuery, Cloud SQL, Cloud Storage, Cloud Composer), Dataplex, Dataform, PostgreSQL, Apache Airflow, Terraform, API integrations, data modelling and warehousing principles Responsibilities Define and maintain target data architecture aligned to strategic objectives Design secure ...

Infrastructure as code: Terraform Containerisation and orchestration: Docker, Kubernetes Source control and CI/CD: GitHub, GitHub Actions Languages: Python Data technologies Workflow orchestration: Apache Airflow Data transformation: dbt Metadata, cataloguing and lineage: DataHub Data access governance: AWS IAM, AWS Lake Formation Development environments : Kubernetes-orchestrated development environments ...

Infrastructure as code: Terraform Containerisation and orchestration: Docker, Kubernetes Source control and CI/CD: GitHub, GitHub Actions Languages: Python Data technologies Workflow orchestration: Apache Airflow Data transformation: dbt Metadata, cataloguing and lineage: DataHub Data access governance: AWS IAM, AWS Lake Formation Development environments : Kubernetes-orchestrated development environments ...

background as a Machine Learning Engineer. Technical Skills: Strong in SQL and Python (Pandas, Scikit-learn, Jupyter, Matplotlib). Data transformation & manipulation : experience with Airflow, DBT and Kubeflow Cloud: Experience with GCP and Vertex AI (developing ML services). Expertise: Solid understanding of computer science fundamentals and time-series ...

technologies including the Python scientific stack (NumPy, Pandas, SciPy, etc.), frameworks such as OpenCV, PyTorch, TensorFlow and JAX, data pipeline tools such as Airflow or Prefect and data processing tools such as Spark or Databricks. The Company The Python Developer will be joining a company whose platform has been ...

technologies including the Python scientific stack (NumPy, Pandas, SciPy, etc.), frameworks such as OpenCV, PyTorch, TensorFlow and JAX, data pipeline tools such as Airflow or Prefect and data processing tools such as Spark or Databricks. The Company The Python Developer will be joining a company whose platform has been ...

directly with clients in a consultative, solution-shaping capacity. * Experience leading or growing technical teams. * Exposure to AWS, automation tools and orchestrators such as Airflow or Dagster. * Understanding of machine learning or predictive analytics, even at a conceptual level. * Experience across multiple industries; familiarity with the music industry ...

outcomes Experience in startups, scale-ups, or consultancies is helpful — but not required. Nice to Have (Not Required) Experience with tools like Databricks, dbt, Airflow, or similar platforms Some exposure to API development or backend systems Experience integrating enterprise systems (ERP, CRM, operational platforms) Familiarity with dashboards or data ...

outcomes Experience in startups, scale-ups, or consultancies is helpful — but not required. Nice to Have (Not Required) Experience with tools like Databricks, dbt, Airflow , or similar platforms Some exposure to API development or backend systems Experience integrating enterprise systems (ERP, CRM, operational platforms) Familiarity with dashboards or data ...

experience with Google Cloud Platform (Big Query, Data form, Cloud Composer). Metadata management and data cataloguing (e.g., DataHub, Alation). Modern orchestration platforms (Airflow, Prefect, Dagster). Cloud-native architectures, including Kubernetes, serverless, and Infrastructure as Code. Data mesh or decentralised data architecture implementations. Model Context Protocol ...

experience with Google Cloud Platform (Big Query, Data form, Cloud Composer). Metadata management and data cataloguing (e.g., DataHub, Alation). Modern orchestration platforms (Airflow, Prefect, Dagster). Cloud-native architectures, including Kubernetes, serverless, and Infrastructure as Code. Data mesh or decentralised data architecture implementations. Model Context Protocol ...