PySpark - Spark Python API
UK

The table below provides summary statistics for permanent job vacancies requiring PySpark skills. It includes a benchmarking guide to the annual salaries offered in vacancies that cited PySpark over the 6 months leading up to 23 May 2025, comparing them to the same period in the previous two years.

6 months to
23 May 2025
Same period 2024 Same period 2023
Rank 438 496 688
Rank change year-on-year +58 +192 +84
Permanent jobs citing PySpark 292 456 274
As % of all permanent jobs advertised in the UK 0.53% 0.46% 0.28%
As % of the Libraries, Frameworks & Software Standards category 3.17% 2.16% 0.92%
Number of salaries quoted 234 190 218
10th Percentile £50,000 £52,500 £50,000
25th Percentile £72,500 £57,500 £62,500
Median annual salary (50th Percentile) £110,000 £70,000 £80,000
Median % change year-on-year +57.14% -12.50% +14.29%
75th Percentile £122,500 £87,500 £105,000
90th Percentile £142,250 £100,000 £160,000
UK excluding London median annual salary £63,750 £75,000 £62,500
% change year-on-year -15.00% +20.00% +13.64%

All Software Libraries and Frameworks
UK

PySpark falls under the Software Libraries and Frameworks category. For comparison with the information above, the following table provides summary statistics for all permanent job vacancies requiring technical specification, industry standards, software libraries and framework skills.

Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 9,223 21,090 29,830
As % of all permanent jobs advertised in the UK 16.90% 21.08% 30.23%
Number of salaries quoted 5,695 15,749 18,355
10th Percentile £38,750 £36,250 £40,500
25th Percentile £51,250 £45,000 £50,500
Median annual salary (50th Percentile) £66,230 £60,000 £67,500
Median % change year-on-year +10.38% -11.11% +5.88%
75th Percentile £87,500 £78,750 £87,500
90th Percentile £111,250 £100,000 £106,250
UK excluding London median annual salary £60,000 £55,000 £57,500
% change year-on-year +9.09% -4.35% +4.55%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Salary Trend

3-month moving average salary quoted in jobs citing PySpark.

Salary trend for PySpark in the UK

PySpark
Salary Histogram

Salary distribution for jobs citing PySpark over the 6 months to 23 May 2025.

Salary histogram for PySpark in the UK

PySpark
Top 16 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the UK over the 6 months to 23 May 2025. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +178 277 £110,000 +57.14% 495
London +130 217 £110,000 +57.14% 135
Work from Home +133 167 £110,000 +62.96% 129
UK excluding London +17 71 £63,750 -15.00% 422
South East +111 18 £52,500 +31.25% 128
Midlands +74 16 £65,000 +4.00% 72
North of England +11 11 £75,000 -9.09% 85
Scotland -22 11 £75,000 +15.38% 20
East of England +47 10 £50,000 -9.09% 57
West Midlands +67 9 £62,500 - 36
East Midlands +39 7 £75,000 - 32
North West -31 5 £75,000 -11.76% 39
South West +99 4 £112,500 +104.55% 58
Yorkshire +59 3 £85,000 +6.25% 35
North East - 3 £62,500 - 11
Wales +36 1 £55,000 +19.41% 13

PySpark
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 143 (48.97%) Apache Airflow
2 125 (42.81%) Apache
3 93 (31.85%) Apache Spark
4 3 (1.03%) SAS
Applications
1 4 (1.37%) Microsoft Excel
2 1 (0.34%) Microsoft Office
Cloud Services
1 184 (63.01%) AWS
2 133 (45.55%) dbt
3 110 (37.67%) Azure
4 86 (29.45%) Serverless
5 85 (29.11%) Amazon ECS
6 76 (26.03%) AWS Lambda
7 69 (23.63%) OpenShift
8 67 (22.95%) Snowflake
9 59 (20.21%) GCP
10 54 (18.49%) Azure Data Factory
11 38 (13.01%) Power Platform
12 32 (10.96%) Dagster
13 29 (9.93%) Azure DevOps
14 27 (9.25%) Azure Synapse Analytics
15 23 (7.88%) Amazon EC2
16 20 (6.85%) Amazon EKS
17 16 (5.48%) Amazon S3
18 12 (4.11%) Amazon Kinesis
18 12 (4.11%) Microsoft Purview
18 12 (4.11%) Power Automate
Communications & Networking
1 1 (0.34%) PuTTY
Database & Business Intelligence
1 138 (47.26%) Amazon Athena
2 133 (45.55%) Apache Iceberg
3 127 (43.49%) Dremio
4 113 (38.70%) Apache Arrow
5 81 (27.74%) Apache Ignite
6 57 (19.52%) Data Lake
7 56 (19.18%) Data Warehouse
8 34 (11.64%) Power BI
9 22 (7.53%) Big Data
10 21 (7.19%) Tableau
11 17 (5.82%) NoSQL
12 14 (4.79%) Delta Lake
12 14 (4.79%) Hadoop
13 13 (4.45%) Amazon Redshift
13 13 (4.45%) Spark SQL
14 10 (3.42%) Metadata
15 8 (2.74%) Parquet
15 8 (2.74%) PostgreSQL
16 7 (2.40%) BigQuery
17 6 (2.05%) Apache Hudi
Development Applications
1 64 (21.92%) Artifactory
2 25 (8.56%) Git
3 8 (2.74%) Jenkins
4 7 (2.40%) Bitbucket
5 5 (1.71%) CircleCI
6 4 (1.37%) GitLab
6 4 (1.37%) Spinnaker
7 1 (0.34%) JIRA
7 1 (0.34%) kappa
7 1 (0.34%) Octopus Deploy
7 1 (0.34%) SpecFlow
General
1 156 (53.42%) Finance
2 127 (43.49%) Front Office
3 77 (26.37%) Banking
4 50 (17.12%) Investment Banking
5 45 (15.41%) Social Skills
6 24 (8.22%) Analytical Skills
7 16 (5.48%) Inclusion and Diversity
8 15 (5.14%) Law
9 13 (4.45%) Public Sector
10 8 (2.74%) Marketing
11 7 (2.40%) Retail
12 3 (1.03%) Games
12 3 (1.03%) Presentation Skills
13 1 (0.34%) Advertising
13 1 (0.34%) Billing
13 1 (0.34%) Influencing Skills
13 1 (0.34%) Legal
13 1 (0.34%) Manufacturing
13 1 (0.34%) Tech for Good
13 1 (0.34%) Telecoms
Job Titles
1 178 (60.96%) Data Engineer
2 118 (40.41%) AWS Engineer
3 108 (36.99%) Python Engineer
4 65 (22.26%) DevOps Engineer
5 63 (21.58%) AWS DevOps
5 63 (21.58%) AWS DevOps Engineer
6 56 (19.18%) Software Engineer
7 50 (17.12%) AWS Software Engineer
7 50 (17.12%) Senior
8 41 (14.04%) Developer
9 34 (11.64%) AWS Developer
10 32 (10.96%) Developer/Software Engineer
10 32 (10.96%) Java Engineer
10 32 (10.96%) Snowflake Developer
10 32 (10.96%) Snowflake Engineer
10 32 (10.96%) Software Developer
11 31 (10.62%) Java Software Engineer
12 24 (8.22%) Python Data Engineer
12 24 (8.22%) Python Developer
12 24 (8.22%) Senior Data Engineer
Libraries, Frameworks & Software Standards
1 88 (30.14%) Kafka
2 63 (21.58%) J2EE
3 50 (17.12%) Pandas
4 35 (11.99%) NumPy
5 32 (10.96%) SciPy
6 18 (6.16%) gRPC
6 18 (6.16%) Protocol Buffers
7 12 (4.11%) scikit-learn
8 11 (3.77%) TensorFlow
9 8 (2.74%) Spark Streaming
10 7 (2.40%) Flask
11 6 (2.05%) PyArrow
12 5 (1.71%) OAuth
13 4 (1.37%) Django
14 3 (1.03%) pytest
14 3 (1.03%) XGBoost
14 3 (1.03%) YAML
15 2 (0.68%) PyTorch
15 2 (0.68%) Smart Contracts
16 1 (0.34%) Shiny
Miscellaneous
1 62 (21.23%) Equities
1 62 (21.23%) Fixed Income
2 50 (17.12%) Derivative
2 50 (17.12%) Foreign Exchange (FX)
2 50 (17.12%) Hedge funds
3 16 (5.48%) Algorithms
4 14 (4.79%) Product Ownership
5 12 (4.11%) Credit Risk
6 6 (2.05%) Data Structures
6 6 (2.05%) Greenfield Project
6 6 (2.05%) Reinsurance
7 5 (1.71%) Distributed Applications
8 4 (1.37%) Cloud Native
8 4 (1.37%) Low-Code
9 3 (1.03%) Life Science
10 2 (0.68%) Blockchain
10 2 (0.68%) Data Fabric
11 1 (0.34%) e-Learning
11 1 (0.34%) Legacy Systems
Operating Systems
1 7 (2.40%) Unix
Processes & Methodologies
1 157 (53.77%) Data Engineering
2 156 (53.42%) Agile
3 147 (50.34%) Software Engineering
4 132 (45.21%) Asset Management
5 112 (38.36%) DevOps
6 107 (36.64%) CI/CD
7 79 (27.05%) Analytics
8 74 (25.34%) ETL
9 69 (23.63%) Data Pipeline
10 65 (22.26%) Open Source
11 63 (21.58%) Site Reliability Engineering
12 57 (19.52%) Data Science
13 50 (17.12%) Data Modelling
14 41 (14.04%) Machine Learning
15 39 (13.36%) Visualisation
16 38 (13.01%) Data Management
16 38 (13.01%) Data Transformation
16 38 (13.01%) Problem-Solving
17 36 (12.33%) Business Intelligence
18 30 (10.27%) Data Analysis
Programming Languages
1 276 (94.52%) Python
2 155 (53.08%) SQL
3 146 (50.00%) Java
4 63 (21.58%) Go
5 21 (7.19%) JavaScript
6 18 (6.16%) Scala
6 18 (6.16%) TypeScript
7 7 (2.40%) T-SQL
8 4 (1.37%) R
9 3 (1.03%) Cypher
10 1 (0.34%) Bash
10 1 (0.34%) Bicep
10 1 (0.34%) C#
10 1 (0.34%) C++
10 1 (0.34%) Kotlin
10 1 (0.34%) PowerShell
10 1 (0.34%) PROC SQL
10 1 (0.34%) VB
Qualifications
1 35 (11.99%) Degree
2 22 (7.53%) Security Cleared
3 20 (6.85%) SC Cleared
4 14 (4.79%) DBS Check
5 11 (3.77%) Azure Certification
5 11 (3.77%) Master's Degree
6 3 (1.03%) AWS Certification
7 2 (0.68%) Computer Science Degree
8 1 (0.34%) Numerate Degree
Quality Assurance & Compliance
1 43 (14.73%) Data Quality
2 10 (3.42%) GDPR
3 8 (2.74%) Accessibility
4 1 (0.34%) QA
System Software
1 84 (28.77%) Docker
2 7 (2.40%) HDFS
3 6 (2.05%) Apache Flume
4 2 (0.68%) Active Directory
Systems Management
1 98 (33.56%) Terraform
2 92 (31.51%) Kubernetes
3 6 (2.05%) Oozie
4 2 (0.68%) Ansible
5 1 (0.34%) Kibana
5 1 (0.34%) logstash
Vendors
1 88 (30.14%) Databricks
2 23 (7.88%) Microsoft
3 7 (2.40%) Cloudera
4 6 (2.05%) Red Hat
5 5 (1.71%) Google
6 2 (0.68%) Foundry
6 2 (0.68%) Informatica
6 2 (0.68%) Palantir
7 1 (0.34%) CheckPoint
7 1 (0.34%) MicroStrategy
7 1 (0.34%) Oracle
7 1 (0.34%) SAP
7 1 (0.34%) Splunk