Period
to 14 December 2019

The following table provides summary statistics for contract job vacancies with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 14 December 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
UK
6 months to
14 Dec 2019
Same period 2018 Same period 2017
Rank 545 751 781
Rank change year-on-year +206 +30 +106
Contract jobs citing PySpark 243 159 117
As % of all contract IT jobs advertised in the UK 0.31% 0.15% 0.11%
As % of the Libraries, Frameworks & Software Standards category 1.02% 0.44% 0.35%
Number of daily rates quoted 174 124 103
UK median daily rate £550 £550 £550
10th Percentile £452 £479 £400
90th Percentile £675 £613 £675
UK excluding London median daily rate £550 £475 £525
% change year-on-year +15.79% -9.52% -4.55%
Number of hourly rates quoted 0 2 0
UK median hourly rate - £24.00 -
UK excluding London median hourly rate - £24.00 -

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 23,871 36,477 33,250
As % of all contract IT jobs advertised in the UK 30.16% 34.76% 32.20%
Number of daily rates quoted 16,465 25,779 23,511
UK median daily rate £490 £475 £450
Median daily rate % change year-on-year +3.16% +5.56% +5.88%
10th Percentile £331 £325 £313
90th Percentile £650 £638 £625
UK excluding London median daily rate £438 £425 £400
% change year-on-year +2.94% +6.25% +3.09%
Number of hourly rates quoted 272 314 363
UK median hourly rate £42.50 £41.52 £40.00
Median hourly rate % change year-on-year +2.37% +3.79% -
10th Percentile £17.50 £19.08 £17.80
90th Percentile £73.75 £60.00 £52.40
UK excluding London median hourly rate £42.00 £41.53 £40.00
% change year-on-year +1.13% +3.83% +3.23%

PySpark
Job Vacancy Trend

Job postings citing PySpark as a percentage of all IT jobs advertised.

Job vacancy trend for PySpark in the UK

PySpark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing PySpark.

Contractor daily rate trend for PySpark in the UK

PySpark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing PySpark over the 6 months to 14 December 2019.

Contractor daily rate histogram for PySpark in the UK

PySpark
Top 14 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the UK over the 6 months to 14 December 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +209 236 £550 - 19
London +104 162 £548 -0.45% 14
UK excluding London +158 79 £550 +15.79% 5
South East +90 38 £585 -10.00% 2
Midlands +80 18 £525 -
West Midlands - 14 £525 -
North of England +52 13 £550 +15.79% 3
Yorkshire +47 9 £550 +15.79% 1
North West - 4 £550 - 2
East Midlands - 4 £475 -
Wales +34 3 £700 +332.10%
East of England - 3 £550 -
Scotland +31 2 - -
South West - 2 £600 -

For the 6 months to 14 December 2019, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for PySpark.

1 218 (89.71%) Python
2 151 (62.14%) Apache Spark
3 107 (44.03%) Azure
4 102 (41.98%) SQL
5 97 (39.92%) AWS
6 94 (38.68%) Hadoop
7 90 (37.04%) Data Science
8 88 (36.21%) Pandas
9 81 (33.33%) Big Data
10 76 (31.28%) Agile Software Development
11 64 (26.34%) Finance
12 62 (25.51%) Analytics
13 59 (24.28%) Machine Learning
14 55 (22.63%) Git (software)
15 54 (22.22%) Data Engineering
16 51 (20.99%) Visualisation
17 48 (19.75%) DevOps
17 48 (19.75%) Apache
18 47 (19.34%) Databricks
19 46 (18.93%) Scala
19 46 (18.93%) NumPy
20 43 (17.70%) ETL
21 41 (16.87%) Java
21 41 (16.87%) Kubernetes
22 40 (16.46%) Data Warehouse
23 39 (16.05%) Continuous Integration
24 38 (15.64%) Artificial Intelligence
25 37 (15.23%) NoSQL
26 36 (14.81%) Docker
27 35 (14.40%) Jenkins

PySpark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 151 (62.14%) Apache Spark
2 48 (19.75%) Apache
3 31 (12.76%) Jupyter
4 21 (8.64%) Elasticsearch
5 19 (7.82%) Apache Airflow
6 11 (4.53%) SAS
7 6 (2.47%) Pachyderm
8 4 (1.65%) Confluence
8 4 (1.65%) OpenStack
9 2 (0.82%) Apache Pig
9 2 (0.82%) Tornado
10 1 (0.41%) Luigi
10 1 (0.41%) Oracle Workflow
Applications
1 2 (0.82%) Microsoft Excel
Business Applications
1 5 (2.06%) SAP Oil and Gas
Cloud Services
1 107 (44.03%) Azure
2 97 (39.92%) AWS
3 24 (9.88%) Azure Machine Learning
4 23 (9.47%) GCP
5 21 (8.64%) Azure Data Factory
6 19 (7.82%) Amazon EMR
7 18 (7.41%) PaaS
8 16 (6.58%) AWS Lambda
8 16 (6.58%) GitHub
8 16 (6.58%) Serverless
9 13 (5.35%) Amazon EKS
9 13 (5.35%) Amazon S3
10 11 (4.53%) Power Platform
11 8 (3.29%) AWS CloudFormation
12 6 (2.47%) Amazon SQS
13 5 (2.06%) Amazon ECS
14 4 (1.65%) Amazon Cognito
14 4 (1.65%) AWS Fargate
14 4 (1.65%) AWS Step Functions
15 3 (1.23%) Amazon Kinesis
Database & Business Intelligence
1 94 (38.68%) Hadoop
2 81 (33.33%) Big Data
3 40 (16.46%) Data Warehouse
4 37 (15.23%) NoSQL
5 29 (11.93%) Apache Hive
6 23 (9.47%) Amazon Athena
7 22 (9.05%) Amazon Redshift
8 19 (7.82%) Relational Database
9 17 (7.00%) RDBMS
9 17 (7.00%) SQL Server
10 16 (6.58%) Geospatial Data
10 16 (6.58%) PostgreSQL
11 13 (5.35%) Azure SQL Database
11 13 (5.35%) GIS
11 13 (5.35%) Spark SQL
12 12 (4.94%) BigQuery
13 11 (4.53%) Power BI
14 10 (4.12%) Data Lake
14 10 (4.12%) Neo4j
15 9 (3.70%) SAS Enterprise Guide
Development Applications
1 55 (22.63%) Git (software)
2 35 (14.40%) Jenkins
3 15 (6.17%) JIRA
4 9 (3.70%) Atlassian Bamboo
4 9 (3.70%) Vagrant
5 6 (2.47%) Visual Studio Team System
6 5 (2.06%) TeamCity
6 5 (2.06%) Travis CI
7 4 (1.65%) Artifactory
7 4 (1.65%) GitLab
7 4 (1.65%) Gradle
7 4 (1.65%) Pylint
7 4 (1.65%) Sonatype Nexus
7 4 (1.65%) Team Foundation Server
8 2 (0.82%) JUnit
9 1 (0.41%) Octopus Deploy
General
1 64 (26.34%) Finance
2 13 (5.35%) Retail
3 7 (2.88%) Retail Banking
4 6 (2.47%) Banking
4 6 (2.47%) Legal
5 5 (2.06%) Investment Banking
5 5 (2.06%) Public Sector
6 4 (1.65%) Financial Institution
7 3 (1.23%) Local Government
8 2 (0.82%) Telecoms
9 1 (0.41%) Automotive
9 1 (0.41%) Games
9 1 (0.41%) Pharmaceutical
Job Titles
1 97 (39.92%) Data Engineer
2 83 (34.16%) Developer
3 66 (27.16%) Python Engineer
4 57 (23.46%) Python Developer
5 43 (17.70%) Data Scientist
6 28 (11.52%) Big Data Engineer
7 21 (8.64%) Hadoop Developer
8 18 (7.41%) AWS Engineer
9 14 (5.76%) Azure Engineer
10 13 (5.35%) Senior Data Scientist
11 11 (4.53%) Senior Data Engineer
12 10 (4.12%) Machine Learning Scientist
13 9 (3.70%) Consultant
14 7 (2.88%) ETL Developer
15 6 (2.47%) AWS Developer
15 6 (2.47%) Java Developer
15 6 (2.47%) Java Hadoop Developer
16 5 (2.06%) Azure Consultant
16 5 (2.06%) Senior Developer
16 5 (2.06%) Senior Python Developer
Libraries, Frameworks & Software Standards
1 88 (36.21%) Pandas
2 46 (18.93%) NumPy
3 34 (13.99%) scikit-learn
4 31 (12.76%) TensorFlow
5 30 (12.35%) Keras
6 18 (7.41%) Flask
7 17 (7.00%) Kafka
8 16 (6.58%) Spark Streaming
9 12 (4.94%) pytest
10 8 (3.29%) Natural Language Processing
10 8 (3.29%) React
10 8 (3.29%) Web Services
11 6 (2.47%) Lambda Expressions
11 6 (2.47%) MLlib
12 5 (2.06%) SparkR
12 5 (2.06%) Spring
13 4 (1.65%) J2EE
13 4 (1.65%) PyTorch
14 3 (1.23%) Apache NiFi
15 2 (0.82%) Anaconda
Miscellaneous
1 13 (5.35%) Algorithms
2 10 (4.12%) Foreign Exchange (FX)
2 10 (4.12%) Lidar
3 9 (3.70%) Analytical Skills
4 8 (3.29%) Smart City
5 5 (2.06%) Linux Command Line
6 4 (1.65%) Greenfield Project
6 4 (1.65%) Online Gambling
7 3 (1.23%) Smart Meter
8 2 (0.82%) Pricing Data
9 1 (0.41%) Clustering
9 1 (0.41%) Data Structures
9 1 (0.41%) Online Games
9 1 (0.41%) Smart Energy
Operating Systems
1 8 (3.29%) Unix
2 7 (2.88%) Linux
3 2 (0.82%) Windows
Processes & Methodologies
1 90 (37.04%) Data Science
2 76 (31.28%) Agile Software Development
3 62 (25.51%) Analytics
4 59 (24.28%) Machine Learning
5 54 (22.22%) Data Engineering
6 51 (20.99%) Visualisation
7 48 (19.75%) DevOps
8 43 (17.70%) ETL
9 39 (16.05%) Continuous Integration
10 38 (15.64%) Artificial Intelligence
11 33 (13.58%) Data Visualisation
12 30 (12.35%) Computer Science
12 30 (12.35%) Deep Learning
13 24 (9.88%) Data Modelling
14 20 (8.23%) Data Architecture
14 20 (8.23%) Mathematics
15 19 (7.82%) Problem-Solving
16 18 (7.41%) B2B
16 18 (7.41%) Data Management
17 17 (7.00%) Business Intelligence
Programming Languages
1 218 (89.71%) Python
2 102 (41.98%) SQL
3 46 (18.93%) Scala
4 41 (16.87%) Java
5 18 (7.41%) Bash Shell
6 16 (6.58%) R
7 10 (4.12%) JavaScript
8 7 (2.88%) PowerShell
9 6 (2.47%) Julia
10 3 (1.23%) T-SQL
11 2 (0.82%) Shell Script
12 1 (0.41%) C#
12 1 (0.41%) PL/SQL
12 1 (0.41%) TypeScript
12 1 (0.41%) VBA
Qualifications
1 18 (7.41%) Degree
2 15 (6.17%) Numerate Degree
2 15 (6.17%) PhD
3 11 (4.53%) Master's Degree
4 4 (1.65%) AWS Certification
5 3 (1.23%) BPSS Clearance
Quality Assurance & Compliance
1 1 (0.41%) QA
System Software
1 36 (14.81%) Docker
2 23 (9.47%) HDFS
3 9 (3.70%) VirtualBox
Systems Management
1 41 (16.87%) Kubernetes
2 24 (9.88%) Ansible
3 22 (9.05%) Terraform
4 19 (7.82%) Hadoop YARN
5 9 (3.70%) Docker Swarm
6 6 (2.47%) Kubeflow
7 5 (2.06%) Oozie
8 3 (1.23%) Grafana
Vendors
1 47 (19.34%) Databricks
2 25 (10.29%) Google
3 22 (9.05%) Tableau
4 20 (8.23%) Cloudera
4 20 (8.23%) Hortonworks
5 18 (7.41%) Oracle
5 18 (7.41%) Splunk
6 17 (7.00%) Informatica
6 17 (7.00%) Palantir
7 16 (6.58%) Talend
8 9 (3.70%) Datactics
8 9 (3.70%) IBM
9 6 (2.47%) MapR
9 6 (2.47%) Microsoft
9 6 (2.47%) SAP
10 5 (2.06%) Alteryx
11 3 (1.23%) Twitter
12 2 (0.82%) StreamSets
12 2 (0.82%) Sybase
13 1 (0.41%) Salesforce.com