1 to 25 of 140 PySpark Jobs in England

Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

needed: • Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory.• Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing.• Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

Strong proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability to design, build, and maintain scalable data pipelines for batch and streaming data using modern ...

added flexibility for diverse migration and integration projects. Prior experience with tools such as MuleSoft, Boomi, Informatica, Talend, SSIS, or custom scripting languages (Python, PySpark, SQL) for data extraction and transformation. Prior experience with Data warehousing and Data modelling (Star Schema or Snowflake Schema). Skilled in security frameworks ...

technical architecture and detailed design. •Expertise in building scalable ingestion frameworks (batch/streaming) using ADF, Databricks or Fabric pipelines. •Strong programming skills in PySpark, SQL, Python. •Experience implementing medallion architecture, delta lakes, and metadata‐driven pipelines. •Hands-on experience with DQ frameworks, validation rules and reconciliation logic. •Ability ...

backgrounds. The technology stack is flexible, but experience in some of the following are nice to have: Data Engineering & Programming: • Python • SQL • Spark/PySpark • Scala • Data modelling and warehousing concepts Cloud & Platforms: • AWS (S3, Redshift, Glue, EMR, Lambda) • Other cloud platforms (Azure/GCP considered) • Data lake ...

Engineer or in a similar role Proven experience with cloud-based data platforms (Azure, AWS, SQL, Snowflake, Springserv); Strong proficiency in Spark SQL and PySpark , including complex transformations Experience building ETL/ELT pipelines using tools such as Azure Data Factory or equivalent Ability to write efficient, reusable scripts ...

culture of learning, quality, and continuous improvement. Essential Skills Strong experience with Azure Databricks and cloud data platforms. Advanced proficiency in Python, PySpark, and SQL. Experience developing Spark/Databricks pipelines. Hands-on experience with structured streaming and event-driven systems. Strong understanding of Lakehouse architecture and best practices. ...

concept initiatives About You To be considered, you should have: Proven experience in a Data Engineering role Strong proficiency in SQL, Python and Spark (PySpark) Experience designing data models and implementing reliable pipeline patterns Knowledge of orchestration tools such as Airflow Experience with low-code or no-code integration ...

Designing and building cloud based data and machine learning pipelines that prepare data for analytics, AI, and product use ·Writing clear, well-structured Python, PySpark, and SQL to transform and validate data from multiple upstream sources ·Taking ownership of data quality, consistency, and reliability across the pipeline lifecycle ·Shaping ...

analytics and visualisation Collaborating with engineers, analysts and stakeholders in Agile delivery teams Tech stack SQL Python Azure or AWS Databricks/Spark/PySpark Data warehouses/lakehouse platforms CI/CD and modern engineering practices We're looking for someone who Has experience building production data pipelines ...

with various databases e.g. MS SQL, Azure Cosmos DB. Skilled at optimizing large and more complicated SQL statements. Proficiency in Python and experience with PySpark Experience of REST API standards. Proficiency in writing clean, readable, and secure code for Azure backend services, i.e. functions, utility functions etc. Experience ...

Senior) of data engineering experience Hands-on experience with Palantir Foundry and/or Foundry Data Platform (FDP) Strong proficiency in: Python SQL Spark (PySpark or SparkSQL) Experience building production data pipelines in cloud environments (AWS, Azure, or GCP) Understanding of data modeling, schema design, and metadata management Familiarity ...

demand delivery Provide technical guidance for team members Providing 2nd or 3rd level technical support About You Experience using SQL, SQL Server DB, Python & PySpark Experience using Azure Data Factory Experience using Data Bricks and Cloudsmith Data Warehousing Experience Project Management Experience The ability to interact with the operational ...

code, with a genuine passion for software engineering in a data context. You'll work across the modern Azure data stack - ADF, ADLS Gen2, PySpark, Delta Lake - with increasing exposure to Microsoft Fabric as the platform matures. You'll collaborate closely with customers and internal teams to ensure data … architectural decisions, and help mentor junior engineers as the team scales. Responsibilities Architect and write production-grade ELT/ETL data pipelines using PySpark and Python within Azure ecosystem. Build custom, reusable data processing frameworks and libraries in Python/Scala to streamline ingestion and transformation tasks across ...

experience required Strong experience in data engineering Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. ...

experience required Strong experience in data engineering Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. ...

Lloyd's of London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record of ownership, delivery and operating effectively in fast-paced environments. What ...

Maintain documentation, lineage, governance standards. Required Skills/Experience The ideal candidate will have the following: Strong SQL; Snowflake ELT; dbt experience. Python/PySpark, ETL/ELT design. Data Vault 2.0 or dimensional modeling. AWS services (S3, Glue, Lambda, Redshift) or GCP equivalents. Experience with CI/ ...

legacy warehouse assets into a modern Azure environment Contribute to cloud architecture decisions, data standards and best-practice engineering patterns Develop reliable Python and PySpark code to support data ingestion, transformation, and end-to-end processing. What you’ll bring Strong hands-on experience across Azure Data Services ...

CDF. High technical proficiency with Cosmos DB SDKs (e.g., .NET, Java, Python, Node.js) Thorough understanding of Spark distributed computing concepts. High technical proficiency in PySpark and Spark Concepts. Experience with concurrency patterns, CLR, and scalable application design. Deep understanding of Azure services including Azure Functions, App Services ...

data pipelines • Engineering ML systems on modern cloud platforms • Working with architects and data engineers on enterprise AI platforms Technology • Python • SQL • Spark/PySpark • TensorFlow, PyTorch or similar frameworks • AWS, Azure or GCP • Databricks or Snowflake • Exposure to GenAI or LLMs Background We’re interested in engineers with ...

Build and operate scalable data pipelines using Microsoft Fabric (OneLake/Delta Lake, Data Factory, Synapse Data Engineering). Develop batch processing solutions using PySpark, Spark SQL, Python, and SQL , with a focus on performance, resilience, and data quality. Support reporting and MI use cases, including transformations and data … Data Engineer or equivalent. Active SC clearance (BPSS minimum). Strong hands-on experience with Microsoft Fabric and Azure data services. Advanced skills in PySpark, Spark SQL, Python, and SQL . Experience delivering data engineering solutions in government or similarly regulated environments. CI/CD, DevOps, and IaC experience ...

Lead Data Engineer - Pyspark/AWS/Python/SAS - Financial Sector As a Lead PySpark Engineer, you will design, develop, and fix complex data processing solutions using PySpark on AWS. You will work hands-on with code, modernising legacy data workflows and supporting large-scale … PySpark migrations. The role requires strong engineering discipline, deep data understanding, and the ability to deliver production-ready data pipelines in a financial services environment. Essential Skills PySpark & Data Engineering Minimum 5+ years of hands-on PySpark experience. SAS to Pyspark migration experience Proven ability ...

practices You have experience with Databricks, ideally also Azure You have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and Pandas You're excited to join a start-up in a role that you can shape and influence You're collaborative with excellent ...