PySpark - Spark Python API
UK

The table below provides summary statistics for permanent job vacancies requiring PySpark skills. It includes a benchmarking guide to the annual salaries offered in vacancies that cited PySpark over the 6 months leading up to 30 April 2025, comparing them to the same period in the previous two years.

6 months to
30 Apr 2025
Same period 2024 Same period 2023
Rank 396 517 711
Rank change year-on-year +121 +194 +78
Permanent jobs citing PySpark 318 415 278
As % of all permanent jobs advertised in the UK 0.61% 0.42% 0.27%
As % of the Libraries, Frameworks & Software Standards category 3.74% 1.97% 0.88%
Number of salaries quoted 261 168 225
10th Percentile £50,000 £52,500 £50,000
25th Percentile £75,563 £59,688 £60,000
Median annual salary (50th Percentile) £110,000 £70,000 £80,000
Median % change year-on-year +57.14% -12.50% +18.52%
75th Percentile £122,500 £88,125 £105,000
90th Percentile £147,500 £100,000 £160,000
UK excluding London median annual salary £75,000 £80,000 £62,500
% change year-on-year -6.25% +28.00% +13.64%

All Software Libraries and Frameworks
UK

PySpark falls under the Software Libraries and Frameworks category. For comparison with the information above, the following table provides summary statistics for all permanent job vacancies requiring technical specification, industry standards, software libraries and framework skills.

Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 8,493 21,046 31,450
As % of all permanent jobs advertised in the UK 16.28% 21.40% 30.24%
Number of salaries quoted 6,474 15,738 19,371
10th Percentile £38,750 £36,250 £41,117
25th Percentile £50,509 £45,000 £50,000
Median annual salary (50th Percentile) £65,000 £60,000 £67,500
Median % change year-on-year +8.33% -11.11% +8.00%
75th Percentile £87,500 £79,900 £87,500
90th Percentile £111,250 £100,000 £106,250
UK excluding London median annual salary £60,000 £55,000 £57,500
% change year-on-year +9.09% -4.35% +4.55%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Salary Trend

3-month moving average salary quoted in jobs citing PySpark.

Salary trend for PySpark in the UK

PySpark
Salary Histogram

Salary distribution for jobs citing PySpark over the 6 months to 30 April 2025.

Salary histogram for PySpark in the UK

PySpark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the UK over the 6 months to 30 April 2025. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +244 299 £110,000 +57.14% 118
London +158 240 £110,000 +57.14% 71
Work from Home +246 193 £110,000 +57.14% 80
UK excluding London +45 73 £75,000 -6.25% 57
South East +131 21 £52,500 +31.25% 11
Scotland -11 15 £77,606 +19.39% 13
Midlands +61 12 £65,000 +3.17% 5
East of England +68 11 £50,000 -9.09% 1
North of England +51 9 £75,000 -11.76% 14
West Midlands +50 7 £58,750 -6.75% 4
South West +118 5 £110,000 +69.23% 12
East Midlands +50 5 £75,000 - 1
North West +4 5 £75,000 -11.76% 9
Yorkshire +65 2 £85,000 +6.25% 1
North East +53 2 £62,500 +13.64% 4

PySpark
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 177 (55.66%) Apache Airflow
2 150 (47.17%) Apache
3 101 (31.76%) Apache Spark
4 6 (1.89%) SAS
5 1 (0.31%) SharePoint
Applications
1 4 (1.26%) Microsoft Excel
2 1 (0.31%) Microsoft Office
Business Applications
1 2 (0.63%) Dynamics CRM
Cloud Services
1 211 (66.35%) AWS
2 166 (52.20%) dbt
3 109 (34.28%) Amazon ECS
4 105 (33.02%) Azure
5 94 (29.56%) Serverless
6 84 (26.42%) AWS Lambda
7 80 (25.16%) OpenShift
7 80 (25.16%) Snowflake
8 79 (24.84%) GCP
9 57 (17.92%) Dagster
10 49 (15.41%) Azure Data Factory
11 39 (12.26%) Amazon EC2
12 34 (10.69%) Amazon EKS
13 31 (9.75%) Power Platform
14 25 (7.86%) Azure DevOps
14 25 (7.86%) Azure Synapse Analytics
15 16 (5.03%) Amazon S3
15 16 (5.03%) Power Automate
16 11 (3.46%) Amazon Kinesis
17 10 (3.14%) Azure Functions
Communications & Networking
1 2 (0.63%) ADSL
1 2 (0.63%) Internet
Database & Business Intelligence
1 169 (53.14%) Amazon Athena
1 169 (53.14%) Apache Iceberg
2 160 (50.31%) Dremio
3 137 (43.08%) Apache Arrow
4 105 (33.02%) Apache Ignite
5 76 (23.90%) Data Lake
6 55 (17.30%) Data Warehouse
7 27 (8.49%) Power BI
8 18 (5.66%) Hadoop
8 18 (5.66%) Tableau
9 17 (5.35%) Big Data
9 17 (5.35%) NoSQL
10 16 (5.03%) Delta Lake
11 14 (4.40%) Spark SQL
12 12 (3.77%) Amazon Redshift
13 9 (2.83%) Apache Hudi
14 8 (2.52%) Parquet
15 7 (2.20%) Metadata
15 7 (2.20%) PostgreSQL
16 6 (1.89%) BigQuery
Development Applications
1 72 (22.64%) Artifactory
2 24 (7.55%) Git
3 10 (3.14%) Bitbucket
4 8 (2.52%) Jenkins
5 5 (1.57%) CircleCI
6 4 (1.26%) GitLab
6 4 (1.26%) Spinnaker
7 1 (0.31%) JIRA
7 1 (0.31%) JMeter
7 1 (0.31%) kappa
7 1 (0.31%) Octopus Deploy
7 1 (0.31%) SpecFlow
General
1 188 (59.12%) Finance
2 160 (50.31%) Front Office
3 100 (31.45%) Banking
4 66 (20.75%) Investment Banking
5 50 (15.72%) Social Skills
6 29 (9.12%) Analytical Skills
7 13 (4.09%) Law
8 9 (2.83%) Inclusion and Diversity
9 8 (2.52%) Public Sector
10 7 (2.20%) Retail
11 6 (1.89%) Marketing
12 5 (1.57%) Presentation Skills
13 3 (0.94%) Advertising
13 3 (0.94%) Games
14 2 (0.63%) Influencing Skills
15 1 (0.31%) Billing
15 1 (0.31%) Legal
15 1 (0.31%) Manufacturing
15 1 (0.31%) Telecoms
Job Titles
1 187 (58.81%) Data Engineer
2 141 (44.34%) AWS Engineer
3 130 (40.88%) Python Engineer
4 73 (22.96%) DevOps Engineer
5 71 (22.33%) AWS DevOps
5 71 (22.33%) AWS DevOps Engineer
6 69 (21.70%) Software Engineer
7 66 (20.75%) AWS Software Engineer
8 59 (18.55%) Senior
9 40 (12.58%) Java Engineer
10 39 (12.26%) Java Software Engineer
11 38 (11.95%) Developer
12 34 (10.69%) AWS Developer
13 32 (10.06%) Developer/Software Engineer
13 32 (10.06%) Python Data Engineer
13 32 (10.06%) Snowflake Developer
13 32 (10.06%) Snowflake Engineer
13 32 (10.06%) Software Developer
14 28 (8.81%) Senior Data Engineer
15 27 (8.49%) Python Software Engineer
Libraries, Frameworks & Software Standards
1 99 (31.13%) Kafka
2 72 (22.64%) Pandas
3 71 (22.33%) J2EE
4 57 (17.92%) NumPy
4 57 (17.92%) SciPy
5 34 (10.69%) gRPC
5 34 (10.69%) Protocol Buffers
6 11 (3.46%) scikit-learn
6 11 (3.46%) TensorFlow
7 9 (2.83%) PyArrow
8 8 (2.52%) Spark Streaming
9 5 (1.57%) OAuth
10 3 (0.94%) Flask
10 3 (0.94%) pytest
10 3 (0.94%) XGBoost
10 3 (0.94%) YAML
11 2 (0.63%) PyTorch
12 1 (0.31%) ARM Templates
12 1 (0.31%) Smart Contracts
12 1 (0.31%) Spring Boot
Miscellaneous
1 84 (26.42%) Equities
1 84 (26.42%) Fixed Income
2 66 (20.75%) Derivative
2 66 (20.75%) Foreign Exchange (FX)
2 66 (20.75%) Hedge funds
3 24 (7.55%) Product Ownership
4 16 (5.03%) Credit Risk
5 15 (4.72%) Algorithms
6 7 (2.20%) Data Structures
6 7 (2.20%) Reinsurance
7 6 (1.89%) Greenfield Project
7 6 (1.89%) Low-Code
8 5 (1.57%) Distributed Applications
9 3 (0.94%) Data Fabric
9 3 (0.94%) Life Science
9 3 (0.94%) Management Information System
10 2 (0.63%) Cloud Native
10 2 (0.63%) Distributed Computing
11 1 (0.31%) Blockchain
11 1 (0.31%) Self-Motivation
Operating Systems
1 9 (2.83%) Unix
Processes & Methodologies
1 185 (58.18%) Agile
2 177 (55.66%) Data Engineering
3 176 (55.35%) Software Engineering
4 165 (51.89%) Asset Management
5 116 (36.48%) DevOps
6 115 (36.16%) CI/CD
7 85 (26.73%) Analytics
8 75 (23.58%) Open Source
9 71 (22.33%) Site Reliability Engineering
10 69 (21.70%) ETL
11 64 (20.13%) Data Pipeline
12 55 (17.30%) Data Science
13 49 (15.41%) Data Modelling
13 49 (15.41%) Migration
14 43 (13.52%) Data Transformation
15 40 (12.58%) Problem-Solving
16 37 (11.64%) Data Management
17 36 (11.32%) Machine Learning
18 32 (10.06%) Visualisation
19 30 (9.43%) Data Governance
Programming Languages
1 302 (94.97%) Python
2 177 (55.66%) Java
3 151 (47.48%) SQL
4 71 (22.33%) Go
5 37 (11.64%) JavaScript
6 34 (10.69%) TypeScript
7 16 (5.03%) Scala
8 4 (1.26%) R
8 4 (1.26%) T-SQL
9 3 (0.94%) Cypher
10 2 (0.63%) Data Analysis Expressions
10 2 (0.63%) VB
11 1 (0.31%) Apex Code
11 1 (0.31%) Bicep
11 1 (0.31%) C#
11 1 (0.31%) C++
11 1 (0.31%) Groovy
11 1 (0.31%) Kotlin
11 1 (0.31%) PowerShell
Qualifications
1 44 (13.84%) Degree
2 21 (6.60%) Security Cleared
3 20 (6.29%) SC Cleared
4 14 (4.40%) DBS Check
5 11 (3.46%) Master's Degree
6 9 (2.83%) Azure Certification
7 4 (1.26%) Computer Science Degree
8 2 (0.63%) AWS Certification
9 1 (0.31%) Numerate Degree
Quality Assurance & Compliance
1 36 (11.32%) Data Quality
2 11 (3.46%) Accessibility
3 9 (2.83%) GDPR
4 4 (1.26%) CRD IV
4 4 (1.26%) IFRS
4 4 (1.26%) IFRS 9
5 1 (0.31%) QA
System Software
1 90 (28.30%) Docker
2 9 (2.83%) HDFS
3 5 (1.57%) Apache Flume
4 2 (0.63%) Active Directory
Systems Management
1 101 (31.76%) Kubernetes
1 101 (31.76%) Terraform
2 5 (1.57%) Oozie
3 2 (0.63%) Ansible
Vendors
1 79 (24.84%) Databricks
2 26 (8.18%) Microsoft
3 9 (2.83%) Red Hat
4 7 (2.20%) Cloudera
5 6 (1.89%) Foundry
5 6 (1.89%) Palantir
6 5 (1.57%) Google
7 2 (0.63%) Apple
8 1 (0.31%) CheckPoint
8 1 (0.31%) Oracle
8 1 (0.31%) Salesforce
8 1 (0.31%) Splunk
8 1 (0.31%) StreamSets