1 to 25 of 140 PySpark Jobs in the UK

platforms on Microsoft Azure. The role involves leading the development of secure, reliable, and efficient data pipelines using Azure Synapse Analytics, Azure Data Factory, PySpark, and Azure Data Lake Storage. The engineer will work closely with cross functional teams to translate business requirements into technical solutions that support analytics ...

contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable: Microsoft Fabric Delta Live Tables Power BI dataset ...

contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable: Microsoft Fabric Delta Live Tables Power BI dataset ...

data engineering team while partnering closely with product, BI, data science, and business stakeholders Experience required: Strong experience with Microsoft Fabric, Delta Lake, PySpark, Python, SQL, and modern ETL/ELT frameworks, alongside CI/CD practices using Azure DevOps or GitHub Hands-on expertise integrating Azure AI/ ...

Warehouses, and real-time analytics solutions in Fabric Strong SQL/T-SQL capability, including optimisation and development of reusable objects. Proficiency in Python (PySpark, Spark SQL beneficial). Experience working with structured/unstructured data, delta lake patterns, partitioning, and scalable pipeline design. Strong understanding of data modelling ...

tooling Key skills Strong experience with Microsoft Fabric (Data Factory, Lakehouse, OneLake, Pipelines, SQL Warehouse) Solid data engineering background using SQL, Python, PySpark, Delta Lake E xperience building and optimising end-to-end data pipelines CI/CD experience for analytics and data platforms (Azure DevOps or GitHub) Exposure ...

Experience* Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala, or SQL. ...

Experience* Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala, or SQL. ...

technical architecture and detailed design. •Expertise in building scalable ingestion frameworks (batch/streaming) using ADF, Databricks or Fabric pipelines. •Strong programming skills in PySpark, SQL, Python. •Experience implementing medallion architecture, delta lakes, and metadata‐driven pipelines. •Hands-on experience with DQ frameworks, validation rules and reconciliation logic. •Ability ...

Proficiency in AWS Glue, Qlik Talend, DBT, Airflow, and modern data integration tools. Excellent knowledge of Python, SQL, PowerQuery (M), and preferably Scala or PySpark Working knowledge with enterprise architecture frameworks (e.g., TOGAF), MLOps, and BI tools like Power BI and QuickSight Experience of generative AI platforms (e.g., Amazon ...

data models and analytics; exposure to predictive modelling and machine learning is a plus. Proficient in SQL and Python, with bonus points for PySpark, SparkSQL, and Git. Skilled in data visualisation with tools such as Tableau or Power BI. Confident writing efficient code and troubleshooting sophisticated queries. Clear ...

Microsoft Fabric including OneLake, Lakehouse, Delta, Data Pipelines, Dataflows and semantic models. • Strong SQL skills including optimisation and performance tuning. • Experience using Python or PySpark for data transformations. • Experience building ETL or ELT pipelines using Fabric Data Pipelines or Azure Data Factory. • Strong understanding of data modelling and modern ...

Demonstrable experience building data pipelines in production environments. Proficiency with Microsoft Fabric, including Data Factory, lakehouses, warehouses, notebooks and semantic models. Strong skills in PySpark, Spark SQL and T-SQL for data transformation. Experience with medallion architecture or equivalent layered data processing patterns. Working knowledge of OneLake, delta tables ...

pipelines and working with lakehouse architectures Knowledge of tools such as Azure Data Factory and/or Fabric Data Pipelines Experience with Python or PySpark for data transformation Understanding of data modelling (dimensional modelling, medallion architecture) Strong communication skills and ability to work with non-technical stakeholders Desirable: Experience ...

added flexibility for diverse migration and integration projects. Prior experience with tools such as MuleSoft, Boomi, Informatica, Talend, SSIS, or custom scripting languages (Python, PySpark, SQL) for data extraction and transformation. Prior experience with Data warehousing and Data modelling (Star Schema or Snowflake Schema). Skilled in security frameworks ...

Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage ). Exposure ...

beneficial: Proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability to design, build, and maintain scalable data pipelines for batch and streaming data using modern ...

team supporting Azure based Production Data & Analytics ETL/integration solutions (Fabric, Databricks, Synapse, Data Factory, Function Apps, Power BI etc) against SLAs Python, PySpark, Scala, Spark SQL Ability to identify and investigate complex problems and implement resolutions Direct contact with customers/end-users on the phone ...

automation and optimization of data workflows Requirements: Experience in data engineering and pipeline development Strong expertise in Azure (Synapse, Data Lake) Proficiency in Python, PySpark, and T-SQL Knowledge of data warehousing, CDC, and incremental processing Familiarity with modern data architectures (e.g., medallion) Preferred Experience with Azure DevOps/ ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data ...

data pipelines using Databricks and Apache Spark Build and optimize data workflows for batch and real-time processing Transform and process large datasets using PySpark, SQL, and Delta Lake Implement data ingestion from multiple sources (APIs, streaming, databases) Your Skills: Python proficiency DBT AWS : Cloud formation , Service Catalog ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

data workflows using Data Lakes and Logic Apps Write and maintain complex T-SQL queries, stored procedures, and optimised database logic Use Python and PySpark for distributed data processing and large-scale transformation within Fabric and cloud BI pipelines Ensure data accuracy, quality, and adherence to BI governance standards ...

best practices are followed Required Skills & Experience: Strong experience with Azure data platform (e.g., Data Factory, Data Lake, Synapse) Hands-on expertise with Databricks (PySpark or Spark) Solid SQL development skills Experience working with Kafka for streaming data pipelines Good understanding of lakehouse architecture, including bronze/silver/ ...

modern data platform development. Deep hands-on experience with Microsoft Fabric, including Lakehouse, Data Warehouse, OneLake, Pipelines and Data Factory. Strong SQL, Spark/PySpark and data modelling capability across analytical and operational datasets. Proven experience delivering complex end-to-end data platforms in Azure or Fabric environments. Strong ...