Period
to 8 August 2020

The following table provides summary statistics for contract job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 8 August 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
UK
6 months to
8 Aug 2020
Same period 2019 Same period 2018
Rank 400 609 815
Rank change year-on-year +209 +206 +28
Contract jobs citing PySpark 149 241 93
As % of all contract jobs advertised in the UK 0.42% 0.26% 0.091%
As % of the Libraries, Frameworks & Software Standards category 1.39% 0.83% 0.26%
Number of daily rates quoted 90 169 60
Median daily rate £525 £550 £550
Median daily rate % change year-on-year -4.55% - -
10th Percentile £380 £485 £458
90th Percentile £663 £625 £675
UK excluding London median daily rate £580 £550 £550
% change year-on-year +5.45% - -
Number of hourly rates quoted 1 1 0
Median hourly rate £35.00 £19.88 -
Median hourly rate % change year-on-year +76.06% - -
UK excluding London median hourly rate - £19.88 -

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 10,686 28,928 35,155
As % of all contract IT jobs advertised in the UK 30.34% 31.73% 34.25%
Number of daily rates quoted 7,105 20,163 24,812
Median daily rate £500 £488 £475
Median daily rate % change year-on-year +2.56% +2.63% +5.56%
10th Percentile £330 £333 £319
90th Percentile £675 £650 £625
UK excluding London median daily rate £450 £438 £420
% change year-on-year +2.86% +4.17% +5.00%
Number of hourly rates quoted 169 319 352
Median hourly rate £35.00 £43.91 £41.00
Median hourly rate % change year-on-year -20.29% +7.10% +2.50%
10th Percentile £22.33 £19.66 £15.24
90th Percentile £63.98 £60.00 £57.50
Median hourly rate £35.00 £42.50 £40.00
% change year-on-year -17.65% +6.25% -

PySpark
Job Vacancy Trend

Job postings citing PySpark as a proportion of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing PySpark.

Daily rate trend for PySpark in the UK

PySpark
Daily Rate Histogram

Daily rate distribution for jobs citing PySpark over the 6 months to 8 August 2020.

Daily rate histogram for PySpark in the UK

PySpark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing PySpark.

Hourly rate trend for PySpark in the UK

PySpark
Top 11 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the UK over the 6 months to 8 August 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +205 136 £525 -4.55% 8
London +134 113 £525 - 7
UK excluding London +221 26 £580 +5.45% 2
North of England +121 18 £578 +5.00% 1
Yorkshire +77 16 £580 +5.45%
Work from Home -9 13 £525 -8.70%
South East +158 3 £600 +11.63%
Wales +35 3 £600 +26.32%
North West +77 2 £525 -7.89% 1
East of England +81 1 £500 -23.08%
Scotland - 1 - - 1

For the 6 months to 8 August 2020, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for PySpark.

1 126 (84.56%) Python
2 88 (59.06%) Apache Spark
3 77 (51.68%) Big Data
4 64 (42.95%) Hadoop
5 59 (39.60%) Scala
6 49 (32.89%) SQL
7 43 (28.86%) Git (software)
8 41 (27.52%) Machine Learning
9 40 (26.85%) AWS
10 38 (25.50%) Data Science
11 37 (24.83%) Analytics
12 36 (24.16%) TensorFlow
12 36 (24.16%) ETL
13 35 (23.49%) Azure
14 34 (22.82%) Agile Software Development
15 33 (22.15%) GCP
16 31 (20.81%) Java
17 30 (20.13%) Apache Hive
17 30 (20.13%) Databricks
18 27 (18.12%) Continuous Integration
18 27 (18.12%) Business Intelligence
19 26 (17.45%) Data Modelling
20 25 (16.78%) Telecoms
21 23 (15.44%) GitLab
21 23 (15.44%) scikit-learn
22 21 (14.09%) Cloudera
23 20 (13.42%) Amazon S3
24 19 (12.75%) TDD
24 19 (12.75%) Apache
24 19 (12.75%) Data Warehouse

PySpark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 88 (59.06%) Apache Spark
2 19 (12.75%) Apache
3 13 (8.72%) Elasticsearch
3 13 (8.72%) SAS
4 10 (6.71%) Apache Airflow
5 9 (6.04%) Apache Pig
6 5 (3.36%) Tornado
7 4 (2.68%) Confluence
7 4 (2.68%) Jupyter
8 1 (0.67%) Apache Solr
Applications
1 1 (0.67%) SPSS
Cloud Services
1 40 (26.85%) AWS
2 35 (23.49%) Azure
3 33 (22.15%) GCP
4 20 (13.42%) Amazon S3
5 16 (10.74%) Azure Data Factory
6 10 (6.71%) Amazon EC2
6 10 (6.71%) GitHub
7 9 (6.04%) Power Platform
8 8 (5.37%) AWS Lambda
9 5 (3.36%) Serverless
10 4 (2.68%) Amazon Kinesis
11 3 (2.01%) Amazon EMR
11 3 (2.01%) Amazon SageMaker
11 3 (2.01%) AWS CloudFormation
11 3 (2.01%) AWS Fargate
11 3 (2.01%) Logic Apps
12 2 (1.34%) Amazon SQS
12 2 (1.34%) Azure Active Directory
12 2 (1.34%) PaaS
13 1 (0.67%) Azure Stack
Communications & Networking
1 3 (2.01%) 5G
2 1 (0.67%) Broadband
2 1 (0.67%) Kerberos
Database & Business Intelligence
1 77 (51.68%) Big Data
2 64 (42.95%) Hadoop
3 30 (20.13%) Apache Hive
4 19 (12.75%) Data Warehouse
5 18 (12.08%) BigQuery
5 18 (12.08%) Data Lake
6 14 (9.40%) HBase
6 14 (9.40%) Impala
7 13 (8.72%) NoSQL
8 11 (7.38%) Azure SQL Database
9 10 (6.71%) Apache Cassandra
9 10 (6.71%) MongoDB
10 9 (6.04%) Amazon Athena
10 9 (6.04%) Power BI
10 9 (6.04%) Presto
11 7 (4.70%) Relational Database
11 7 (4.70%) Sqoop
12 6 (4.03%) Data Mining
12 6 (4.03%) DataStage
12 6 (4.03%) Redis
Development Applications
1 43 (28.86%) Git (software)
2 23 (15.44%) GitLab
3 16 (10.74%) Jenkins
4 10 (6.71%) Bitbucket
5 9 (6.04%) TeamCity
6 8 (5.37%) Artifactory
6 8 (5.37%) JIRA
7 5 (3.36%) Hudson
7 5 (3.36%) ProC
8 4 (2.68%) Visual Studio Team System
9 3 (2.01%) CircleCI
9 3 (2.01%) ERwin
9 3 (2.01%) Maven
10 1 (0.67%) Cucumber
10 1 (0.67%) JUnit
General
1 25 (16.78%) Telecoms
2 18 (12.08%) Finance
3 8 (5.37%) Banking
3 8 (5.37%) Marketing
4 7 (4.70%) Retail
5 5 (3.36%) Legal
6 3 (2.01%) Public Sector
7 2 (1.34%) Games
7 2 (1.34%) Investment Banking
8 1 (0.67%) Advertising
8 1 (0.67%) Electronics
8 1 (0.67%) Manufacturing
8 1 (0.67%) Pharmaceutical
8 1 (0.67%) Retail Banking
Job Titles
1 76 (51.01%) Data Engineer
2 35 (23.49%) Python Engineer
3 26 (17.45%) Big Data Engineer
4 22 (14.77%) Azure Engineer
5 18 (12.08%) Data Scientist
6 17 (11.41%) Developer
7 13 (8.72%) Python Developer
8 12 (8.05%) Machine Learning Engineer
9 11 (7.38%) Architect
10 10 (6.71%) Technical Architect
11 7 (4.70%) AWS Engineer
12 6 (4.03%) Senior Data Engineer
13 5 (3.36%) Big Data Developer
13 5 (3.36%) Cloud Engineer
13 5 (3.36%) Consultant
13 5 (3.36%) Senior Architect
13 5 (3.36%) Senior Data Scientist
13 5 (3.36%) Senior Technical Architect
13 5 (3.36%) Software Engineer
14 4 (2.68%) Data Consultant
Libraries, Frameworks & Software Standards
1 36 (24.16%) TensorFlow
2 23 (15.44%) scikit-learn
3 18 (12.08%) H2O
3 18 (12.08%) Kafka
4 11 (7.38%) Pandas
5 10 (6.71%) JSON
5 10 (6.71%) MLlib
6 8 (5.37%) ADO
6 8 (5.37%) Anaconda
7 6 (4.03%) Keras
7 6 (4.03%) Spark Streaming
8 5 (3.36%) CSS
8 5 (3.36%) CSS3
8 5 (3.36%) Elastic Stack
8 5 (3.36%) Flask
8 5 (3.36%) HTML
8 5 (3.36%) HTML5
8 5 (3.36%) Samza
8 5 (3.36%) SciPy
8 5 (3.36%) Twisted
Miscellaneous
1 16 (10.74%) Algorithms
2 14 (9.40%) Analytical Skills
3 13 (8.72%) Data Structures
4 8 (5.37%) Enterprise Software
4 8 (5.37%) PMI
5 7 (4.70%) FMCG
6 5 (3.36%) Credit Risk
6 5 (3.36%) Greenfield Project
7 4 (2.68%) Analytical Mindset
8 2 (1.34%) Pricing Data
9 1 (0.67%) NHS
Operating Systems
1 1 (0.67%) Linux
Processes & Methodologies
1 41 (27.52%) Machine Learning
2 38 (25.50%) Data Science
3 37 (24.83%) Analytics
4 36 (24.16%) ETL
5 34 (22.82%) Agile Software Development
6 27 (18.12%) Business Intelligence
6 27 (18.12%) Continuous Integration
7 26 (17.45%) Data Modelling
8 19 (12.75%) TDD
9 17 (11.41%) DevOps
10 16 (10.74%) Artificial Intelligence
11 15 (10.07%) Continuous Deployment
11 15 (10.07%) Scrum
11 15 (10.07%) Visualisation
12 14 (9.40%) BDD
13 13 (8.72%) Software Engineering
14 11 (7.38%) SDLC
14 11 (7.38%) Use Case
15 10 (6.71%) Data Engineering
15 10 (6.71%) Translating Business Requirements
Programming Languages
1 126 (84.56%) Python
2 59 (39.60%) Scala
3 49 (32.89%) SQL
4 31 (20.81%) Java
5 13 (8.72%) R
6 11 (7.38%) Ruby
7 5 (3.36%) JavaScript
8 1 (0.67%) Bash Shell
Qualifications
1 13 (8.72%) Security Cleared
2 11 (7.38%) SC Cleared
3 8 (5.37%) PMI Certification
4 7 (4.70%) Degree
5 4 (2.68%) Microsoft Certification
5 4 (2.68%) Numerate Degree
6 2 (1.34%) Master's Degree
6 2 (1.34%) PhD
7 1 (0.67%) AWS Certification
7 1 (0.67%) DBS Check
7 1 (0.67%) Postgraduate
Quality Assurance & Compliance
1 1 (0.67%) QA
System Software
1 11 (7.38%) Docker
2 7 (4.70%) HDFS
3 2 (1.34%) Active Directory
3 2 (1.34%) Apache Flume
4 1 (0.67%) Apache ZooKeeper
Systems Management
1 12 (8.05%) Kubernetes
2 11 (7.38%) Terraform
3 9 (6.04%) Ansible
4 5 (3.36%) Hadoop YARN
4 5 (3.36%) Oozie
5 1 (0.67%) Ambari
Vendors
1 30 (20.13%) Databricks
2 21 (14.09%) Cloudera
3 13 (8.72%) Google
4 12 (8.05%) Hortonworks
5 11 (7.38%) Microsoft
6 5 (3.36%) Informatica
6 5 (3.36%) StreamSets
7 4 (2.68%) MapR
7 4 (2.68%) Tableau
8 2 (1.34%) Oracle
8 2 (1.34%) SAP
9 1 (0.67%) Confluent
9 1 (0.67%) Splunk
9 1 (0.67%) Talend