Period
to 21 October 2020

The following table provides summary statistics for permanent job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited PySpark over the 6 months to 21 October 2020 with a comparison to the same period in the previous 2 years.

PySpark - Spark Python API
UK
6 months to
21 Oct 2020
Same period 2019 Same period 2018
Rank 606 986 1103
Rank change year-on-year +380 +117 +86
Permanent jobs citing PySpark 74 95 71
As % of all permanent jobs advertised in the UK 0.15% 0.069% 0.043%
As % of the Libraries, Frameworks & Software Standards category 0.35% 0.18% 0.11%
Number of salaries quoted 66 72 63
Median annual salary £63,750 £72,500 £80,000
Median salary % change year-on-year -12.07% -9.38% +37.93%
10th Percentile £50,000 £50,000 £48,500
90th Percentile £105,000 £95,000 £97,500
UK excluding London median annual salary £55,000 £56,675 £53,750
% change year-on-year -2.96% +5.44% +27.98%

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Permanent vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 21,419 51,371 65,427
As % of all permanent IT jobs advertised in the UK 43.82% 37.51% 40.00%
Number of salaries quoted 18,272 42,213 54,108
Median annual salary £57,500 £55,000 £51,273
Median salary % change year-on-year +4.55% +7.27% +2.55%
10th Percentile £35,000 £32,500 £31,250
90th Percentile £93,750 £87,500 £82,500
UK excluding London median annual salary £50,000 £45,000 £45,000
% change year-on-year +11.11% - +5.88%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Salary Trend

3-month moving average salary quoted in jobs citing PySpark.

Salary trend for PySpark in the UK

PySpark
Salary Histogram

Salary distribution for jobs citing PySpark over the 6 months to 21 October 2020.

Salary histogram for PySpark in the UK

PySpark
Top 9 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing PySpark within the UK over the 6 months to 21 October 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +361 71 £62,500 -13.79% 17
London +262 59 £65,000 -16.13% 14
UK excluding London +311 13 £55,000 -2.96% 4
North of England +169 8 £60,000 -11.11%
North West +127 7 £60,000 -11.11%
Work from Home - 5 £53,750 - 3
Scotland +94 3 £67,500 +19.10% 1
South West +116 2 £51,250 +5.13% 1
Yorkshire +124 1 - -

For the 6 months to 21 October 2020, IT jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for PySpark.

1 73 (98.65%) Python
2 44 (59.46%) SQL
3 43 (58.11%) AWS
3 43 (58.11%) Apache Spark
4 39 (52.70%) ETL
5 33 (44.59%) Amazon Redshift
6 32 (43.24%) Data Science
7 29 (39.19%) Data Warehouse
8 26 (35.14%) Business Intelligence
9 25 (33.78%) Tableau
9 25 (33.78%) Hadoop
9 25 (33.78%) Amazon S3
9 25 (33.78%) Azure
9 25 (33.78%) Amazon Athena
9 25 (33.78%) Power Platform
9 25 (33.78%) Power BI
9 25 (33.78%) GCP
10 24 (32.43%) Databricks
10 24 (32.43%) Big Data
10 24 (32.43%) Finance
11 23 (31.08%) Machine Learning
11 23 (31.08%) Data Lake
12 22 (29.73%) Analytics
13 21 (28.38%) Snowflake Schema
14 19 (25.68%) Data Modelling
14 19 (25.68%) Visualisation
15 18 (24.32%) Amazon RDS
15 18 (24.32%) Data Engineering
16 17 (22.97%) DevOps
16 17 (22.97%) Looker

PySpark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 43 (58.11%) Apache Spark
2 14 (18.92%) Apache
3 8 (10.81%) SAS
4 1 (1.35%) Apache Airflow
4 1 (1.35%) Confluence
4 1 (1.35%) Elasticsearch
4 1 (1.35%) Luigi
Applications
1 3 (4.05%) Microsoft Excel
Cloud Services
1 43 (58.11%) AWS
2 25 (33.78%) Amazon S3
2 25 (33.78%) Azure
2 25 (33.78%) GCP
2 25 (33.78%) Power Platform
3 7 (9.46%) Amazon Kinesis
3 7 (9.46%) Amazon SQS
3 7 (9.46%) SaaS
3 7 (9.46%) Serverless
4 6 (8.11%) AWS Lambda
5 4 (5.41%) Amazon ElastiCache
5 4 (5.41%) AWS CloudFormation
6 3 (4.05%) Amazon SageMaker
6 3 (4.05%) AWS CodeBuild
6 3 (4.05%) AWS CodeDeploy
6 3 (4.05%) AWS CodePipeline
6 3 (4.05%) Azure Data Factory
6 3 (4.05%) GitHub
6 3 (4.05%) Google Sheets
7 2 (2.70%) Amazon ECS
Communications & Networking
1 4 (5.41%) Internet
Database & Business Intelligence
1 33 (44.59%) Amazon Redshift
2 29 (39.19%) Data Warehouse
3 25 (33.78%) Amazon Athena
3 25 (33.78%) Hadoop
3 25 (33.78%) Power BI
4 24 (32.43%) Big Data
5 23 (31.08%) Data Lake
6 21 (28.38%) Snowflake Schema
7 18 (24.32%) Amazon RDS
8 17 (22.97%) Looker
9 15 (20.27%) BigQuery
10 12 (16.22%) NoSQL
11 11 (14.86%) PostgreSQL
12 9 (12.16%) Apache Hive
12 9 (12.16%) MySQL
13 8 (10.81%) Azure SQL Data Warehouse
13 8 (10.81%) Azure SQL Database
14 7 (9.46%) RDBMS
14 7 (9.46%) Relational Database
14 7 (9.46%) Spark SQL
Development Applications
1 8 (10.81%) Jenkins
2 4 (5.41%) Git (software)
3 3 (4.05%) GitLab
3 3 (4.05%) Team Foundation Server
4 1 (1.35%) CircleCI
4 1 (1.35%) GoCD
4 1 (1.35%) JIRA
4 1 (1.35%) Visual Studio
General
1 24 (32.43%) Finance
2 13 (17.57%) Banking
3 2 (2.70%) Marketing
4 1 (1.35%) Dutch Language
4 1 (1.35%) Games
4 1 (1.35%) Investment Banking
4 1 (1.35%) Manufacturing
4 1 (1.35%) Retail
4 1 (1.35%) Telecoms
Job Titles
1 46 (62.16%) Data Engineer
2 13 (17.57%) Python Engineer
3 9 (12.16%) AWS Engineer
3 9 (12.16%) Developer
4 8 (10.81%) Data Manager
4 8 (10.81%) Data Scientist
4 8 (10.81%) Senior Manager
5 7 (9.46%) Data Science Manager
5 7 (9.46%) Senior Data Engineer
6 6 (8.11%) Data Science Director
7 5 (6.76%) PHP Software Engineer
7 5 (6.76%) Software Engineer
8 3 (4.05%) Cloud Developer
8 3 (4.05%) Cloud Engineer
8 3 (4.05%) Full Stack Developer
8 3 (4.05%) PHP Developer
8 3 (4.05%) Python Developer
9 2 (2.70%) Azure Engineer
9 2 (2.70%) Big Data Engineer
9 2 (2.70%) Senior Data Scientist
Libraries, Frameworks & Software Standards
1 15 (20.27%) Kafka
2 12 (16.22%) NumPy
2 12 (16.22%) Pandas
3 10 (13.51%) TensorFlow
4 9 (12.16%) Keras
4 9 (12.16%) PyTorch
4 9 (12.16%) React
4 9 (12.16%) scikit-learn
4 9 (12.16%) SciPy
5 8 (10.81%) Matplotlib
5 8 (10.81%) Natural Language Processing
5 8 (10.81%) NetworkX
5 8 (10.81%) Shiny
5 8 (10.81%) SpaCy
5 8 (10.81%) XGBoost
6 4 (5.41%) jQuery
6 4 (5.41%) REST
6 4 (5.41%) Spark Streaming
7 3 (4.05%) CSS
7 3 (4.05%) HTML
Miscellaneous
1 13 (17.57%) Greenfield Project
2 12 (16.22%) Algorithms
3 9 (12.16%) Derivative
4 7 (9.46%) Analytical Skills
5 4 (5.41%) Internet of Things
6 1 (1.35%) Data Structures
6 1 (1.35%) FMCG
6 1 (1.35%) Reinsurance
Operating Systems
1 7 (9.46%) Linux
2 3 (4.05%) Unix
3 1 (1.35%) Red Hat Enterprise Linux
Processes & Methodologies
1 39 (52.70%) ETL
2 32 (43.24%) Data Science
3 26 (35.14%) Business Intelligence
4 23 (31.08%) Machine Learning
5 22 (29.73%) Analytics
6 19 (25.68%) Data Modelling
6 19 (25.68%) Visualisation
7 18 (24.32%) Data Engineering
8 17 (22.97%) DevOps
9 15 (20.27%) Mathematics
10 14 (18.92%) Decision-Making
11 12 (16.22%) Computer Science
11 12 (16.22%) Data Transformation
11 12 (16.22%) Use Case
12 11 (14.86%) ETL Development
12 11 (14.86%) Mentoring
12 11 (14.86%) Strategic Roadmap
13 10 (13.51%) Artificial Intelligence
14 9 (12.16%) Logistic Regression
14 9 (12.16%) Statistics
Programming Languages
1 73 (98.65%) Python
2 44 (59.46%) SQL
3 16 (21.62%) R
4 14 (18.92%) Scala
5 10 (13.51%) JavaScript
6 9 (12.16%) PHP
7 8 (10.81%) MATLAB
8 5 (6.76%) Java
9 3 (4.05%) C#
10 1 (1.35%) PHP 7
10 1 (1.35%) T-SQL
Qualifications
1 14 (18.92%) Degree
2 9 (12.16%) Computer Science Degree
3 8 (10.81%) Master's Degree
Quality Assurance & Compliance
1 3 (4.05%) QA
2 1 (1.35%) GDPR
System Software
1 12 (16.22%) Docker
2 4 (5.41%) Apache Flume
3 2 (2.70%) HDFS
Systems Management
1 3 (4.05%) Ambari
1 3 (4.05%) Hadoop YARN
1 3 (4.05%) Kubernetes
1 3 (4.05%) Oozie
2 1 (1.35%) Ansible
Vendors
1 25 (33.78%) Tableau
2 24 (32.43%) Databricks
3 14 (18.92%) Cloudera
4 7 (9.46%) Microsoft
4 7 (9.46%) Vertica
5 5 (6.76%) Oracle
6 4 (5.41%) Google
7 3 (4.05%) Guidewire
7 3 (4.05%) Hortonworks
7 3 (4.05%) Informatica
7 3 (4.05%) MuleSoft
7 3 (4.05%) StreamSets
8 2 (2.70%) Splunk
9 1 (1.35%) Confluent
9 1 (1.35%) MapR