1 to 25 of 33 Apache Airflow Jobs in Scotland

Data Analytics & Business Intelligence (BI) Experience 5-10 years Location United Kingdom - Glasgow, London Skills Apache Airflow, AWS, Data Lake/Lakehouse, Databricks Certified (Data Engineer, ML), DBT, Python, SQL We are seeking a highly skilled Сonsultant Data Engineer to be the driving force behind this massive architectural … work closely with System Analysts, Automation Engineers, and Data Architects to convert hundreds of legacy scripts into modern dbt models, orchestrate them via Astronomer (Airflow), and establish governed warehousing in Databricks. Requirements Required Skills & Technologies Cloud Platforms: Hands-on engineering experience within the AWS ecosystem (S3, IAM, Glue ...

processes. Technical Skills Programming: Proficiency in Python, Java, Scala, or similar languages. Big Data Technologies: Hands-on experience with big data tools e.g. (Databricks, Apache Spark, Hadoop). Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Expertise in relational databases (e.g. postgres … server) Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines for deployment. Experience 2+ ...

have Programming: Proficiency in Python, Java, Scala, or similar languages for data processing. Big Data Technologies: Hands-on experience with tools such as Databricks, Apache Spark, and Hadoop. Cloud Platforms: Familiarity with AWS, Azure, GCP, or other cloud ecosystems for data engineering tasks. Database Management: Expertise in relational databases … e.g. Postgres, SQL Server). Data Integration Tools: Knowledge of platforms like Airflow, Apache NiFi, or Talend. Data Storage and Modelling: Experience with data warehousing tools (e.g. Snowflake, Redshift, BigQuery) and schema design. Version Control and CI/CD: Familiarity with Git, Docker, and CI/CD pipelines ...

years in a team leadership or mentorship role.Strong technical background in building and maintaining data pipelines and platforms using tools like Spark, dbt, Airflow, Kafka, or Databricks.Proficiency in one or more programming languages (e.g., Python, Scala, or Java) and SQL.Experience with cloud data platforms (e.g., AWS, GCP, or Azure ...

working knowledge Nice-to-have Databricks → AWS (or cross-platform) migration experience — the single strongest signal Parity/statistical testing methodology Data orchestration (Airflow, dbt, Step Functions) Data governance & compliance (PII/GDPR); regulated industry background (iGaming strongly preferred, but FS, banking considered) Soft skills Comfortable working ...

/ELT, and various database technologies. Technical Versatility: Fluency in data processing languages (Python, SQL, PySpark) and openness to tech stacks such as SSIS, Airflow, Spark, Databricks, Fabric. Cloud Capability: Experience with at least one major cloud provider (AWS, Azure, or Google Cloud) and curiosity to learn others. Bridging ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real‐time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

writing automation scripts or custom AI agents using node or python. Data Engineering (30%): Experience deploying and maintaining data orchestration toolsets (e.g., dbt, Apache Airflow), ETL/ELT methodologies, real-time streaming (e.g., AWS Kinesis, Apache Kafka), Vector databases, and RAG architectures. Observability & FinOps: Experience implementing modern ...

supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred) Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred) Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting ...

implementing agentic guardrails. Data Engineering (30%): Practical experience implementing components of data pipelines, including real-time streaming tools (AWS Kinesis, Kafka), data orchestration (dbt, Airflow), and managing vector databases for RAG architectures. Observability & Cost Management: SRE/Platform experience with the practical application of real-time monitoring and cloud ...

Application Resiliency, and Security. Preferred qualifications, capabilities, and skills Production experience with Databricks (Delta Lake, Unity Catalog, MLflow, Databricks Jobs) and workflow orchestration with Apache Airflow. Strong working knowledge of both supervised and unsupervised techniques, including tree-based and kernel methods (gradient boosting, random forest, SVM) and anomaly-detection ...

/CD, Application Resiliency, and Security.Preferred qualifications, capabilities, and skillsProduction experience with Databricks (Delta Lake, Unity Catalog, MLflow, Databricks Jobs) and workflow orchestration with Apache Airflow.Experience with evaluation frameworks and approaches (golden datasets, LLM-as-judge, human-in-the-loop review, red teaming).Solid grounding in data pre-processing ...

continuous delivery (CI/CD) pipelines, containerisation, and workflow orchestration.Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.)Knowledge of programming languages (e.g. Java, Python, SQL)Hands-on experience with SQL/NoSQL database design.Degree in STEM, or similar field ...

delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM, or similar ...

delivery (CI/CD) pipelines, containerisation, and workflow orchestration. Familiar with ETL/ELT frameworks, and experienced with Big Data Processing Tools (e.g. Spark, Airflow, Hive, etc.) Knowledge of programming languages (e.g. Java, Python, SQL) Hands-on experience with SQL/NoSQL database design. Degree in STEM, or similar ...

Working knowledge of Kubernetes/EKS, comfortable deploying to and reasoning about containerised workloads in a managed K8s environment. Experience orchestrating pipelines with Dagster, Airflow, Prefect, or similar. Solid grasp of ETL concepts and production‐grade pipeline design. Beneficial Skills Interest in chemistry, life sciences, robotics, or manufacturing. Hands ...

data quality, security, and accessibility. Develop reusable, production-grade ETL/ELT frameworks and data processing solutions. Implement orchestration workflows using AWS Step Functions, Airflow, and other automation tools. Develop and maintain cloud infrastructure using AWS CloudFormation. Collaborate with business stakeholders to understand requirements and translate them into scalable …/ML use cases. Required Skills & Experience Strong hands-on experience in Data Engineering with delivery of production-grade solutions. Expertise in PySpark, Apache Spark, Python, and SQL. Strong experience designing and optimizing complex data pipelines and ETL/ELT frameworks. Hands-on experience with AWS services including ...

CycleSolid understanding of agile methodologies such as CI/CD, Applicant Resiliency, and Security, Jules, Jenkins, Git/StashPreferred qualifications, capabilities, and skillsKnowledge of Airflow, Data Mesh architectureExperience with data analysisJ.P. Morgan is a global leader in financial services, providing strategic advice and products to the world’s most ...

agile methodologies such as CI/CD, Applicant Resiliency, and Security, Jules, Jenkins, Git/Stash Preferred Qualifications, Capabilities, And Skills Knowledge of Airflow, Data Mesh architecture Experience with data analysis About Us J.P. Morgan is a global leader in financial services, providing strategic advice and products ...

agile methodologies such as CI/CD, Applicant Resiliency, and Security, Jules, Jenkins, Git/Stash Preferred qualifications, capabilities, and skills Knowledge of Airflow, Data Mesh architecture Experience with data analysis #J-18808-Ljbffr ...

data storytelling. A Data Architect: You have a high bar for data quality and have experience working within modern data stacks (e.g BigQuery, dbt, Airflow) What’s In It For You A range of flexible working options to dedicate time to what matters to you Competitive compensation package + ...

data processing. Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning. Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar). Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.). Strong understanding of data quality, security … with business and technical stakeholders. Preferred qualifications, capabilities, and skills Knowledge of regulatory reporting and financial data aggregation techniques. Experience with data orchestration tools (Airflow, Step Functions, etc.). Databricks or AWS certifications. Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning). Experience with Parquet ...

large-scale data processing.Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning.Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar).Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.).Strong understanding of data quality, security, and lineage best … collaborate effectively with business and technical stakeholders.Preferred qualifications, capabilities, and skillsKnowledge of regulatory reporting and financial data aggregation techniques.Experience with data orchestration tools (Airflow, Step Functions, etc.).Databricks or AWS certifications.Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning).Experience with Parquet, JSON, CSV, Avro, Delta ...

data processing. Proficiency in SQL, Python, and PySpark, with experience in query optimization and performance tuning. Hands-on experience with data lake platforms (Databricks, Apache Spark, or similar). Experience with AWS cloud services (S3, ECS, SNS/SQS, Lambda, etc.). Strong understanding of data quality, security … with business and technical stakeholders. Preferred qualifications, capabilities, and skills Knowledge of regulatory reporting and financial data aggregation techniques. Experience with data orchestration tools (Airflow, Step Functions, etc.). Databricks or AWS certifications. Demonstrated proficiency in cloud-native development (e.g., cloud, artificial intelligence, machine learning). Experience with Parquet ...

data engineering Experience in large-scale data processing, microservices, API design, Kafka, Redis, MemCached, observability tools (Dynatrace, Splunk, Grafana), and orchestration frameworks (Airflow, Temporal) Advanced working knowledge of relational and NoSQL databases, vector stores, data lake architectures, and data governance Practical cloud-native experience (AWS, Azure, or GCP) Ability … hands-on experience with Spark/PySpark and other big data processing technologies Expertise in open-source table formats and catalog services such as Apache Iceberg Experience with LLM orchestration frameworks and model serving infrastructure or managed endpoints (AWS Bedrock, Azure OpenAI) Familiarity with emerging technologies in data engineering ...