23 of 23 Apache Airflow Jobs in the South East

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality checks into pipelines. ...

Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments CI/CD experience with a focus on production ownership and reliability ...

Requirements Passion for building scalable, resilient cloud data platforms. Strong experience with Databricks or Snowflake on AWS. Proven Python skills, including Spark and Airflow expertise. Advanced SQL skills and end-to-end data modelling experience. Experience building batch and real-time data integrations. Hands-on CICD skills with ...

need: Strong Python, PySpark, SQL and Databricks or Snowflake experience AWS cloud data engineering expertise Experience building scalable batch and real-time data pipelines Airflow, CI/CD, Git and modern engineering practices Knowledge of Terraform or similar IaC tools Experience with Docker and Kubernetes is beneficial Exposure ...

distributed processing frameworks like SparkSQL Experience with CI/CD, version control (e.g. Git, Jenkins) and pipeline orchestration tools such as Airflow Due to the volume of applications received for positions, it will not be possible to respond to all applications and only applicants who are considered suitable ...

roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data foundations ...

experience includes:• Databricks or Snowflake, ideally within AWS• Python, PySpark and SQL• ETL pipelines and data transformation• End-to-end data modelling• DBT, SparkSQL, Airflow, Git, Jenkins or similar tools• Data quality, governance and performance optimisation• Working well with technical and non-technical teamsThe senior role needs someone with ...

Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

approaches, fit to the use case Build and maintain transformation layers in dbt (or equivalent), with appropriate testing, documentation and lineage Operate orchestration tooling (Airflow, Dagster, Prefect or equivalent) and the workflows around it Manage cloud data warehouses and lakehouses (Snowflake, BigQuery, Redshift, Synapse, Databricks) Implement data quality, testing … Python skills (or Scala/Java where relevant) for pipeline development and tooling Hands-on experience with dbt or equivalent transformation tooling Orchestration experience (Airflow, Dagster, Prefect or equivalent) Production experience with at least one major cloud warehouse/lakehouse (Snowflake, BigQuery, Redshift, Synapse, Databricks) Cloud platform experience ...

orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic and transparency of underlying code Owned a project across a whole ...

similar hands-on leadership role Strong hands-on expertise in Python Extensive experience with Databricks, including workflows, notebooks and Delta Lake Strong experience with Apache Spark/PySpark Experience building and optimising enterprise-scale ETL pipelines Strong understanding of modern data architecture and Lakehouse concepts Advanced SQL and distributed … other highly regulated enterprise environments would be highly advantageous Desirable experience includes: Databricks certifications Kafka or Structured Streaming CI/CD and DevOps practices Airflow or Databricks Workflows Delta Lake optimisation techniques Docker and Kubernetes Exposure to machine learning pipelines Job Offer Competitive day rate ...

supporting modern cloud-based data platforms. Understanding of data modelling and data transformation principles. Ability to work within collaborative engineering teams. Desirable Skills Airflow Databricks AI-focused development experience Semantic layer experience Exposure to Azure-based environments and integrations. Reasonable Adjustments: Respect and equality are core values to us. ...

platform as the business continues to invest Tech Environment: Azure Python SQL Server Azure Data Services Data Warehousing Data Engineering & ETL Workflow orchestration (Airflow, Luigi or similar) You'll probably be someone who: Has experience leading or mentoring Data Engineers Still enjoys writing code and solving engineering problems ...

Python and SQL, familiarity with Scala or R advantageous Experience with data orchestration and workflow management tooling such as Azure Data Factory or Airflow Please see full job description attached. What you can expect from us We believe doing your best work starts with feeling supported both professionally ...

abstraction layers Model the cost, feasibility and risk of different LLMs and implementation approaches THE STACK AWS, Bedrock, LangGraph, LangChain, Python, FastAPI, PostgreSQL, Snowflake, Airflow, React, Next.js, Kubernetes. YOUR EXPERIENCE 15+ years in enterprise software engineering with a background in product engineering organisations or technology companies Deeply technical: recent ...

cloud-native architectures, with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats ...

Python, Bash, Ansible, Puppet) Familiarity with containerisation (Docker, Kubernetes) Nice to have: Experience with CI/CD pipelines (GitLab, Jenkins) Exposure to data tooling (Airflow, NiFi, Elasticsearch) Experience with Cloudera/Hadoop ecosystems (Spark, Hive, HDFS) Why join? Work on modern data platforms at scale Collaborate with data scientists ...

designing and developing scalable data pipelines Proficiency in SQL, Python, Databricks and/or Azure technologies Experience working with tools such as Git and Airflow Strong troubleshooting skills with the ability to test, maintain and improve data platforms A proven track record of delivering high-quality, dependable data solutions ...

We are recruiting for an Analytical Engineer on behalf of one of the UK's most recognised and celebrated personal lines insurers. The successful candidate will be working alongside a talented team of Data & AI ...

infrastructure: AWS and/or GCP, Kubernetes APIs and services: REST, gRPC Distributed systems: event-driven architectures, including Kafka Orchestration Frameworks: LangGraph, LangChain, AirFlow, etc Activities Integration of commercial and open-source LLMs into agentic workflows Agent and orchestration frameworks such as LangChain, LlamaIndex, Semantic Kernel, or CrewAI Model ...

This is a great opportunity to contribute greatly to an extremely engineering-focused organization and further your experience across technologies such as Cloud Composer, Airflow, Dataflow, Spark, Terraform, Google Cloud Platform, etc; You will own engineering efforts to provide partner teams within the organisation with data sources that build … opportunity for the role to go permanent. What You'll Do: Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow; Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality ...

manage data models, schema structures and data reconciliation processes across multiple source systems. Build, maintain and improve orchestration frameworks using tools such as Airflow and dbt. Develop APIs from scratch to support data access, integration and product functionality. Work closely with AI engineers, machine learning specialists and software developers … schema management principles. Experience creating and maintaining APIs from the ground up, rather than simply integrating with existing APIs. Hands-on experience using Airflow, dbt or similar orchestration and transformation tools. Experience working with large, complex and varied datasets. Not the Right Fit If Your background is heavily focused ...