21 of 21 PySpark Jobs in Scotland

e.g. Power BI) Deep understanding of data wareho using concepts, ETL/ELT pipelines and dimensional modelling Proficiency in advanced programming languages (Python/PySpark, SQL) Experience in data pipeline orchestration (e.g. Airflow, Data Factory) Familiarity with DevOps and CI/CD practices (Git, Azure DevOps etc) Ability ...

years of hands-on experience in enterprise data modelling using tools such as ERwin, ER/Studio, or Power Designer Proven experience with Databricks, PySpark, Python, SQL, and modern data platform architectures including data lakes and data warehouses Knowledge of batch and streaming ingestion, CI/CD tools (Azure ...

years of hands-on experience in enterprise data modelling using tools such as ERwin, ER/Studio, or Power Designer Proven experience with Databricks, PySpark, Python, SQL, and modern data platform architectures including data lakes and data warehouses Knowledge of batch and streaming ingestion, CI/CD tools (Azure ...

years of hands-on experience in enterprise data modelling using tools such as ERwin, ER/Studio, or Power Designer Proven experience with Databricks, PySpark, Python, SQL, and modern data platform architectures including data lakes and data warehouses Knowledge of batch and streaming ingestion, CI/CD tools (Azure ...

practices for data governance and security. Required Skills & Experience 5–10 years in data engineering roles, ideally within consultancy environments. Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice ...

practices for data governance and security. Required Skills & Experience 5–10 years in data engineering roles, ideally within consultancy environments. Strong proficiency in: Databricks (PySpark) Azure Data Factory (ADF) SQL (complex queries and optimisation) Azure Data Services (Data Lake, Synapse, etc.) Excellent communication and stakeholder management skills. Nice ...

Looking For: Proven hands-on experience in data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like ...

Looking For: Proven hands-on experience in data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like ...

Looking For: Proven hands-on experience in data engineering projects Good hands-on experience of designing, implementing, debugging ETL pipeline Expertise in Python, PySpark and SQL languages Expertise with Spark and Airflow Experience of designing data pipelines using cloud native services on AWS Extensive knowledge of AWS services like ...

Required Skills & Experience Proven experience in MLOps or DevOps roles within machine learning environments Strong programming skills in Python, with hands-on experience in PySpark and SQL Deep understanding of ML lifecycle management and CI/CD best practices Familiarity with cloud-native ML platforms and scalable deployment strategies ...

have delivered ML techniques to existing pricing models. Financial Services or Insurance experience would be preferable but not limited to. Full Tech Stack - Databricks, PySpark, Microsoft Azure, Azure ML, Azure Stream Analytics, Cognitive Services, Event Hubs, Synapse, Data Factory. This is a fantastic role for an experienced Machine Learning ...

have delivered ML techniques to existing pricing models. Financial Services or Insurance experience would be preferable but not limited to. Full Tech Stack - Databricks, PySpark, Microsoft Azure, Azure ML, Azure Stream Analytics, Cognitive Services, Event Hubs, Synapse, Data Factory. This is a fantastic role for an experienced Machine Learning ...

have delivered ML techniques to existing pricing models. Financial Services or Insurance experience would be preferable but not limited to. Full Tech Stack - Databricks, PySpark, Microsoft Azure, Azure ML, Azure Stream Analytics, Cognitive Services, Event Hubs, Synapse, Data Factory. This is a fantastic role for an experienced Machine Learning ...

Hiring – Senior Data Engineer (PySpark & AWS) Salary: 300 outside IR35 Location: Remote in UK We are looking for an experienced and highly skilled Senior Data Engineer to join our growing data engineering team. This role is ideal for a passionate engineer who thrives in building scalable data platforms, designing … ensure data is reliable, secure, and available in real time and batch processing environments. Key Responsibilities Design, build, and maintain scalable data pipelines using PySpark and Python for high-volume, high-velocity data processing. Develop and manage ETL/ELT workflows, ensuring data accuracy, consistency, and performance. Orchestrate complex ...

TensorFlow Serving, TorchServe, ONNX, and TensorRT. Build and manage ML pipelines using MLflow, Kubeflow, and Azure ML Pipelines. Work with large-scale data using PySpark and integrate models into production environments. Monitor model performance and retrain as needed to ensure accuracy and efficiency. Collaborate with cross-functional teams ...

Sales, Customer Care and Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze ? Silver ? Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee … Databricks: Lakeflow Declarative Pipelines, Lakeflow Workflows, Unity Catalog, Delta Lake Azure: ADLS Gen2, Data Factory, Event Hubs (optional), Key Vault, private endpoints Languages: PySpark, Spark SQL, Python, Git DevOps: Azure DevOps Repos & Pipelines, CI/CD Analytics: Power BI, Fabric What We're Looking For Experience Commercial and proven ...

Sales, Customer Care and Logistics. What You'll Do Lakehouse Engineering (Azure + Databricks) Build and maintain scalable ELT pipelines using Lakeflow Declarative Pipelines, PySpark and Spark SQL. Work within a Medallion architecture (Bronze Silver Gold) to deliver reliable, high-quality datasets. Ingest data from multiple sources including ChargeBee … Databricks: Lakeflow Declarative Pipelines, Lakeflow Workflows, Unity Catalog, Delta Lake Azure: ADLS Gen2, Data Factory, Event Hubs (optional), Key Vault, private endpoints Languages: PySpark, Spark SQL, Python, Git DevOps: Azure DevOps Repos & Pipelines, CI/CD Analytics: Power BI, Fabric What We're Looking For Experience Commercial and proven ...

analysis, governance, reporting and analytics. Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. The salary on offer for this Data Business Analyst Role is up to £120,000 + Bonus and Benefits … define functional and system requirements Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. Essential Skills for the Data Business Analyst: Senior IT Business Analyst/Data Analyst with experience of a Commodities ...

analysis, governance, reporting and analytics. Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. The salary on offer for this Data Business Analyst Role is up to £120,000 + Bonus and Benefits … define functional and system requirements Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. Essential Skills for the Data Business Analyst: Senior IT Business Analyst/Data Analyst with experience of a Commodities ...

analysis, governance, reporting and analytics. Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. The salary on offer for this Data Business Analyst Role is up to £120,000 + Bonus and Benefits … define functional and system requirements Hands-on delivery experience working with data, analysis and BI tools e.g. SQL, PowerQuery, Excel VBA, Python (e.g. Pandas, PySpark), DataBricks, Power BI, Tableau etc. Essential Skills for the Data Business Analyst: Senior IT Business Analyst/Data Analyst with experience of a Commodities ...

open-source data validation frameworks to ensure data accuracy, schema integrity, and quality across ingestion and transformation pipelines Test and validate data pipelines and PySpark notebooks developed by Data Engineers, ensuring they meet quality, reliability, and validation standards Defining and standardizing monitoring, logging, alerting, and KPIs/SLAs across … setting up Alert rules, building dashboards using data queried (KQL) from Log Analytics. Experience with Fabric Data Factory, Azure Data Factory, Synapse pipelines, and PySpark notebooks Hands-on experience calling REST/OData APIs for validating data. Experience writing SQL and scripts for programmatically doing data validations and reconciliation ...