1 to 25 of 136 PySpark Jobs in the UK

platforms on Microsoft Azure. The role involves leading the development of secure, reliable, and efficient data pipelines using Azure Synapse Analytics, Azure Data Factory, PySpark, and Azure Data Lake Storage. The engineer will work closely with cross functional teams to translate business requirements into technical solutions that support analytics ...

contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable: Microsoft Fabric Delta Live Tables Power BI dataset ...

contribute to team capability building Technical Stack Core: Azure Data Factory Azure Synapse Analytics Azure Data Lake Storage Gen2 SQL Server Databricks Enhancements: Python (PySpark, Pandas) CI/CD (Azure DevOps) Infrastructure as Code (Terraform, Bicep) REST APIs GitHub Actions Desirable: Microsoft Fabric Delta Live Tables Power BI dataset ...

Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

business teams while implementing data governance, version control, and quality assurance processes to ensure data integrity and usability. About you: Experience building and deploying PySpark notebooks in data warehousing environments, along with strong SQL skills for data transformation and modelling. Solid understanding of relational and dimensional data modelling techniques ...

needed: • Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory.• Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing.• Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability to design, build, and maintain scalable data pipelines for batch and streaming data using modern ...

technical architecture and detailed design. •Expertise in building scalable ingestion frameworks (batch/streaming) using ADF, Databricks or Fabric pipelines. •Strong programming skills in PySpark, SQL, Python. •Experience implementing medallion architecture, delta lakes, and metadata‐driven pipelines. •Hands-on experience with DQ frameworks, validation rules and reconciliation logic. •Ability ...

working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python, Scala ...

Experience * Proven experience delivering enterprise-scale Databricks solutions end-to-end. * Strong background in Lakehouse Architecture, including structured and unstructured data. * Expertise in Spark, PySpark, Delta Lake, and Databricks workflows. * Experience building scalable ETL/ELT pipelines, including Delta Live Tables. * Strong programming skills in Python, Scala, or SQL. ...

Experience * Proven experience delivering enterprise-scale Databricks solutions end-to-end. * Strong background in Lakehouse Architecture, including structured and unstructured data. * Expertise in Spark, PySpark, Delta Lake, and Databricks workflows. * Experience building scalable ETL/ELT pipelines, including Delta Live Tables. * Strong programming skills in Python, Scala, or SQL. ...

Demonstrable experience building data pipelines in production environments. Proficiency with Microsoft Fabric, including Data Factory, lakehouses, warehouses, notebooks and semantic models. Strong skills in PySpark, Spark SQL and T-SQL for data transformation. Experience with medallion architecture or equivalent layered data processing patterns. Working knowledge of OneLake, delta tables ...

Engineer or in a similar role Proven experience with cloud-based data platforms (Azure, AWS, SQL, Snowflake, Springserv); Strong proficiency in Spark SQL and PySpark , including complex transformations Experience building ETL/ELT pipelines using tools such as Azure Data Factory or equivalent Ability to write efficient, reusable scripts ...

added flexibility for diverse migration and integration projects. Prior experience with tools such as MuleSoft, Boomi, Informatica, Talend, SSIS, or custom scripting languages (Python, PySpark, SQL) for data extraction and transformation. Prior experience with Data warehousing and Data modelling (Star Schema or Snowflake Schema). Skilled in security frameworks ...

team supporting Azure based Production Data & Analytics ETL/integration solutions (Fabric, Databricks, Synapse, Data Factory, Function Apps, Power BI etc) against SLAs Python, PySpark, Scala, Spark SQL Ability to identify and investigate complex problems and implement resolutions Direct contact with customers/end-users on the phone ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

concept initiatives About You To be considered, you should have: Proven experience in a Data Engineering role Strong proficiency in SQL, Python and Spark (PySpark) Experience designing data models and implementing reliable pipeline patterns Knowledge of orchestration tools such as Airflow Experience with low-code or no-code integration ...

Lead PySpark Engineer As a Lead PySpark Engineer , you will design, develop, and optimise complex data processing solutions on AWS. You will work hands-on with PySpark, modernise Legacy data workflows, and support large-scale SAS-to-PySpark migration programmes. This role requires strong engineering discipline … deep data expertise, and the ability to deliver production-ready data pipelines within a financial services environment. Skill Profile: PySpark - P3 (Advanced) AWS - P3 (Advanced) SAS - P1 (Foundational) Key Responsibilities Technical Delivery Design, develop, and fix complex PySpark code for ETL/ELT and data-mart workloads. Convert ...

analytics and visualisation Collaborating with engineers, analysts and stakeholders in Agile delivery teams Tech stack SQL Python Azure or AWS Databricks/Spark/PySpark Data warehouses/lakehouse platforms CI/CD and modern engineering practices We're looking for someone who Has experience building production data pipelines ...

with various databases e.g. MS SQL, Azure Cosmos DB. Skilled at optimizing large and more complicated SQL statements. Proficiency in Python and experience with PySpark Experience of REST API standards. Proficiency in writing clean, readable, and secure code for Azure backend services, i.e. functions, utility functions etc. Experience ...

Maintain documentation, lineage, governance standards. Required Skills/Experience The ideal candidate will have the following: Strong SQL; Snowflake ELT; dbt experience. Python/PySpark, ETL/ELT design. Data Vault 2.0 or dimensional modeling. AWS services (S3, Glue, Lambda, Redshift) or GCP equivalents. Experience with CI/ ...

legacy warehouse assets into a modern Azure environment Contribute to cloud architecture decisions, data standards and best-practice engineering patterns Develop reliable Python and PySpark code to support data ingestion, transformation, and end-to-end processing. What you’ll bring Strong hands-on experience across Azure Data Services ...

demand delivery Provide technical guidance for team members Providing 2nd or 3rd level technical support About You Experience using SQL, SQL Server DB, Python & PySpark Experience using Azure Data Factory Experience using Data Bricks and Cloudsmith Data Warehousing Experience Project Management Experience The ability to interact with the operational ...

code, with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … architectural decisions, and help mentor junior engineers as the team scales. Responsibilities Architect and write production-grade ELT/ETL data pipelines using PySpark and Python within Azure ecosystem. Build custom, reusable data processing frameworks and libraries in Python/Scala to streamline ingestion and transformation tasks across ...