PySpark - Spark Python API
UK

The following table provides summary statistics for permanent job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 29 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
29 May 2024
Same period 2023 Same period 2022
Rank 500 696 737
Rank change year-on-year +196 +41 +75
Permanent jobs citing PySpark 457 269 578
As % of all permanent jobs advertised in the UK 0.44% 0.28% 0.35%
As % of the Libraries, Frameworks & Software Standards category 2.14% 0.92% 0.84%
Number of salaries quoted 195 213 185
10th Percentile £52,500 £50,000 £47,500
25th Percentile £58,750 £62,500 £54,000
Median annual salary (50th Percentile) £70,000 £80,000 £70,000
Median % change year-on-year -12.50% +14.29% +7.69%
75th Percentile £90,000 £105,000 £80,000
90th Percentile £100,000 £160,000 £82,500
UK excluding London median annual salary £77,500 £65,000 £55,000
% change year-on-year +19.23% +18.18% -4.35%

All Software Libraries and Frameworks
UK

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 21,396 29,097 68,823
As % of all permanent jobs advertised in the UK 20.72% 29.95% 41.31%
Number of salaries quoted 15,919 18,040 29,839
10th Percentile £36,250 £40,810 £37,500
25th Percentile £45,000 £50,500 £47,500
Median annual salary (50th Percentile) £60,000 £67,500 £65,000
Median % change year-on-year -11.11% +3.85% +18.18%
75th Percentile £78,750 £87,500 £82,500
90th Percentile £100,000 £106,250 £100,000
UK excluding London median annual salary £55,000 £57,500 £55,000
% change year-on-year -4.35% +4.55% +10.00%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Salary Trend

3-month moving average salary quoted in jobs citing PySpark.

Salary trend for PySpark in the UK

PySpark
Salary Histogram

Salary distribution for jobs citing PySpark over the 6 months to 29 May 2024.

Salary histogram for PySpark in the UK

PySpark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the UK over the 6 months to 29 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +67 261 £70,000 -12.82% 76
UK excluding London +131 250 £77,500 +19.23% 39
London +120 187 £70,000 -24.32% 42
Scotland +79 175 £65,000 -3.70% 6
Work from Home +83 161 £70,000 -17.65% 48
North of England +50 43 £85,000 +30.77% 11
North West +28 37 £85,000 +6.25% 6
Midlands -8 14 £62,500 +13.64% 9
South West -18 12 £65,000 -20.00% 6
West Midlands +8 10 £62,500 +25.00% 4
Yorkshire +71 6 £80,000 +64.10% 4
East Midlands -26 4 - - 5
Northern Ireland - 3 £60,000 - 2
South East -37 2 £40,000 -37.25% 5
East of England -3 1 £55,000 - 1

PySpark
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 168 (36.76%) Apache Spark
2 127 (27.79%) Apache Airflow
3 30 (6.56%) Apache
4 4 (0.88%) Luigi
4 4 (0.88%) SAS
5 1 (0.22%) Apache Pig
5 1 (0.22%) SharePoint
Applications
1 7 (1.53%) Microsoft Excel
Business Applications
1 4 (0.88%) Primavera
2 1 (0.22%) SAP ERP
Cloud Services
1 238 (52.08%) AWS
2 225 (49.23%) Azure
3 89 (19.47%) GCP
4 56 (12.25%) Azure Data Factory
5 53 (11.60%) Azure Synapse Analytics
6 52 (11.38%) Azure DevOps
7 40 (8.75%) Power Platform
8 28 (6.13%) GitHub
9 21 (4.60%) Azure Cosmos DB
10 16 (3.50%) Snowflake
11 14 (3.06%) Serverless
12 12 (2.63%) Amazon S3
12 12 (2.63%) AWS Lambda
12 12 (2.63%) dbt
12 12 (2.63%) Power Automate
12 12 (2.63%) PowerApps
13 11 (2.41%) GitHub Actions
14 10 (2.19%) AWS CloudFormation
15 8 (1.75%) Amazon Kinesis
15 8 (1.75%) SaaS
Communications & Networking
1 2 (0.44%) Network Security
Database & Business Intelligence
1 135 (29.54%) Big Data
2 120 (26.26%) Data Lake
3 78 (17.07%) Data Warehouse
4 60 (13.13%) SQL Server
5 47 (10.28%) Tableau
6 40 (8.75%) Power BI
6 40 (8.75%) Qlik Sense
7 33 (7.22%) Delta Lake
8 23 (5.03%) Azure SQL Database
9 20 (4.38%) Data Vault
10 18 (3.94%) NoSQL
10 18 (3.94%) Spark SQL
11 16 (3.50%) Hadoop
12 10 (2.19%) Amazon Redshift
13 9 (1.97%) Amazon Athena
13 9 (1.97%) SQL Server Integration Services
14 8 (1.75%) Apache Hive
14 8 (1.75%) MySQL
15 7 (1.53%) Data Mining
15 7 (1.53%) PostgreSQL
Development Applications
1 72 (15.75%) Git
2 22 (4.81%) JIRA
3 17 (3.72%) Jenkins
4 11 (2.41%) MLflow
5 9 (1.97%) CircleCI
6 4 (0.88%) Octopus Deploy
7 3 (0.66%) Team Foundation Server
7 3 (0.66%) TeamCity
8 2 (0.44%) GitLab
8 2 (0.44%) JMeter
8 2 (0.44%) JUnit
8 2 (0.44%) MSTest
8 2 (0.44%) NUnit
8 2 (0.44%) Postman
8 2 (0.44%) SpecFlow
9 1 (0.22%) Selenium
9 1 (0.22%) SoapUI
9 1 (0.22%) Subversion
General
1 234 (51.20%) Finance
2 191 (41.79%) Inclusion and Diversity
3 180 (39.39%) Law
4 107 (23.41%) Marketing
5 97 (21.23%) Social Skills
6 82 (17.94%) Legal
7 26 (5.69%) Analytical Skills
8 18 (3.94%) Banking
9 12 (2.63%) Retail
10 7 (1.53%) Retail Banking
11 6 (1.31%) Front Office
12 5 (1.09%) Influencing Skills
12 5 (1.09%) Publishing
13 2 (0.44%) Manufacturing
13 2 (0.44%) Presentation Skills
13 2 (0.44%) Public Sector
14 1 (0.22%) Automotive
14 1 (0.22%) Organisational Skills
14 1 (0.22%) Pharmaceutical
Job Titles
1 288 (63.02%) Data Engineer
2 182 (39.82%) Senior
3 178 (38.95%) Software Engineer
4 120 (26.26%) Python Engineer
5 108 (23.63%) Lead
6 99 (21.66%) Python Software Engineer
7 85 (18.60%) Lead Software Engineer
8 79 (17.29%) Big Data Engineer
8 79 (17.29%) Big Data Software Engineer
9 76 (16.63%) Senior Data Engineer
10 57 (12.47%) Data Scientist
11 53 (11.60%) Lead Data Engineer
12 50 (10.94%) Senior Data Scientist
13 47 (10.28%) Azure Engineer
14 40 (8.75%) Azure Data Engineer
14 40 (8.75%) Lead Big Data Engineer
15 34 (7.44%) Senior Software Engineer
16 21 (4.60%) Python Data Engineer
17 9 (1.97%) Consultant
17 9 (1.97%) Data Consultant
Libraries, Frameworks & Software Standards
1 69 (15.10%) Pandas
2 57 (12.47%) Kafka
3 47 (10.28%) scikit-learn
4 45 (9.85%) TensorFlow
5 9 (1.97%) AWS CDK
6 8 (1.75%) NumPy
6 8 (1.75%) SciPy
7 7 (1.53%) Apache Avro
7 7 (1.53%) Apache Beam
7 7 (1.53%) JSON
7 7 (1.53%) Spark Streaming
8 4 (0.88%) SQLAlchemy
9 3 (0.66%) SOAP
10 2 (0.44%) MDX
10 2 (0.44%) Plotly
10 2 (0.44%) PyTorch
11 1 (0.22%) ARM Templates
11 1 (0.22%) pytest
11 1 (0.22%) React
11 1 (0.22%) Swagger
Miscellaneous
1 90 (19.69%) Algorithms
2 54 (11.82%) Cloud Native
3 39 (8.53%) Onboarding
4 36 (7.88%) Reinsurance
5 26 (5.69%) Data Structures
5 26 (5.69%) Public Cloud
6 16 (3.50%) Greenfield Project
7 9 (1.97%) Product Ownership
8 7 (1.53%) Distributed Applications
8 7 (1.53%) Distributed Systems
9 5 (1.09%) Low-Code
10 4 (0.88%) Self-Motivation
10 4 (0.88%) Team-Oriented Environment
11 2 (0.44%) Hybrid Cloud
11 2 (0.44%) Management Information System
11 2 (0.44%) PropTech
12 1 (0.22%) Blockchain
12 1 (0.22%) Distributed Computing
12 1 (0.22%) Legacy Applications
Operating Systems
1 2 (0.44%) Linux
2 1 (0.22%) Ubuntu
Processes & Methodologies
1 271 (59.30%) Agile
2 207 (45.30%) CI/CD
3 197 (43.11%) Software Engineering
4 180 (39.39%) Machine Learning
4 180 (39.39%) SDLC
5 172 (37.64%) Operational Stability
6 143 (31.29%) Data Engineering
7 142 (31.07%) ETL
8 139 (30.42%) Artificial Intelligence
9 130 (28.45%) Community of Practice
10 102 (22.32%) Decision-Making
11 101 (22.10%) Data Science
12 100 (21.88%) Visualisation
13 87 (19.04%) Mentoring
14 79 (17.29%) Data Pipeline
15 78 (17.07%) Continuous Improvement
15 78 (17.07%) Data Modelling
16 76 (16.63%) Analytics
17 73 (15.97%) Risk Management
18 72 (15.75%) Microservices
Programming Languages
1 407 (89.06%) Python
2 249 (54.49%) SQL
3 89 (19.47%) Java
4 18 (3.94%) T-SQL
5 14 (3.06%) R
6 13 (2.84%) Scala
7 7 (1.53%) C#
7 7 (1.53%) Data Analysis Expressions
8 2 (0.44%) PL/SQL
9 1 (0.22%) Julia
9 1 (0.22%) Shell Script
Qualifications
1 73 (15.97%) Degree
2 49 (10.72%) Master's Degree
3 47 (10.28%) DBS Check
4 16 (3.50%) Computer Science Degree
5 15 (3.28%) Microsoft Certification
6 9 (1.97%) Azure Certification
7 4 (0.88%) CIMA
7 4 (0.88%) PhD
8 2 (0.44%) SC Cleared
8 2 (0.44%) Security Cleared
9 1 (0.22%) Postgraduate
Quality Assurance & Compliance
1 80 (17.51%) Data Quality
2 6 (1.31%) QA
System Software
1 7 (1.53%) Docker
1 7 (1.53%) HDFS
2 1 (0.22%) Apache Flume
Systems Management
1 65 (14.22%) Kubernetes
2 21 (4.60%) Terraform
Vendors
1 216 (47.26%) Databricks
2 39 (8.53%) Alteryx
3 36 (7.88%) Microsoft
4 32 (7.00%) Informatica
4 32 (7.00%) Oracle
5 10 (2.19%) Cloudera
6 8 (1.75%) Google
7 5 (1.09%) OpenAI
7 5 (1.09%) SAP
7 5 (1.09%) SnapLogic
8 4 (0.88%) Splunk
9 3 (0.66%) Teradata
10 2 (0.44%) ServiceNow
11 1 (0.22%) Facebook
11 1 (0.22%) Marketo
11 1 (0.22%) Salesforce
11 1 (0.22%) Twitter