Period
to 23 June 2018

The following table provides summary statistics for contract job vacancies advertised in England with a requirement for PySpark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited PySpark over the 6 months to 23 June 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

PySpark - Spark Python API
UK > England
6 months to
23 Jun 2018
Same period 2017 Same period 2016
Rank 767 784 868
Rank change year-on-year +17 +84 -
Contract jobs citing PySpark 75 22 1
As % of all contract IT jobs advertised in England 0.082% 0.026% 0.001%
As % of the Libraries, Frameworks & Software Standards category 0.24% 0.081% 0.004%
Number of daily rates quoted 56 23 1
England median daily rate £575 £525 £448
Median daily rate % change year-on-year +9.52% +17.32% -
10th Percentile £439 £430 £424
90th Percentile £763 £645 £471
UK median daily rate £560 £513 £448
% change year-on-year +9.27% +14.53% -
Number of hourly rates quoted 0 0 0
England median hourly rate - - -
UK median hourly rate - - -

PySpark is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in England with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
England
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 30,899 27,188 28,358
As % of all contract IT jobs advertised in England 33.76% 32.22% 28.87%
Number of daily rates quoted 21,915 18,875 20,046
England median daily rate £475 £450 £425
Median daily rate % change year-on-year +5.56% +5.88% +6.25%
10th Percentile £315 £306 £291
90th Percentile £631 £613 £600
UK median daily rate £470 £445 £425
% change year-on-year +5.62% +4.71% +6.25%
Number of hourly rates quoted 334 354 387
England median hourly rate £42.38 £41.25 £38.00
Median hourly rate % change year-on-year +2.75% +8.55% +8.57%
10th Percentile £16.25 £17.53 £14.25
90th Percentile £65.83 £53.75 £56.13
UK median hourly rate £42.25 £41.00 £37.50
% change year-on-year +3.05% +9.33% +7.14%

PySpark
Job Vacancy Trend in England

Job postings citing PySpark as a percentage of all IT jobs advertised in England.

Job vacancy trend for PySpark in England

PySpark
Contractor Daily Rate Trend in England

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing PySpark in England.

Contractor daily rate trend for PySpark in England

PySpark
Contractor Daily Rate Histogram in England

The daily rate distribution of IT jobs citing PySpark in England over the 6 months to 23 June 2018.

Contractor daily rate histogram for PySpark in England

PySpark
Contract Job Locations in England

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing PySpark within the England region over the 6 months to 23 June 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London -28 47 £575 +4.55% 6
North of England - 15 £488 - 1
Yorkshire - 14 £475 - 1
South East -13 13 £600 +26.32%
North West - 1 £550 -
PySpark
UK

For the 6 months to 23 June 2018, IT contractor jobs citing PySpark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the England region with a requirement for PySpark.

1 66 (88.00%) Python
2 48 (64.00%) Data Science
3 45 (60.00%) Machine Learning
4 39 (52.00%) Apache Spark
5 34 (45.33%) Amazon AWS
6 30 (40.00%) Hadoop
7 27 (36.00%) Analytics
8 26 (34.67%) SQL
9 25 (33.33%) Big Data
10 22 (29.33%) Pandas
11 21 (28.00%) Jupyter
12 19 (25.33%) Deep Learning
12 19 (25.33%) Scala
13 18 (24.00%) Apache Hive
14 17 (22.67%) Mathematics
15 16 (21.33%) Computer Science
15 16 (21.33%) Artificial Intelligence
16 15 (20.00%) Agile Software Development
17 14 (18.67%) TDD
17 14 (18.67%) Linux
17 14 (18.67%) Apache
17 14 (18.67%) scikit-learn
18 13 (17.33%) Geospatial Data
18 13 (17.33%) TensorFlow
18 13 (17.33%) Web Services
18 13 (17.33%) MongoDB
18 13 (17.33%) Microsoft Azure
18 13 (17.33%) Visualisation
18 13 (17.33%) Continuous Integration
18 13 (17.33%) Apache Pig

PySpark
Co-occurring IT Skills in England by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 39 (52.00%) Apache Spark
2 21 (28.00%) Jupyter
3 14 (18.67%) Apache
4 13 (17.33%) Apache Pig
5 5 (6.67%) Elasticsearch
6 2 (2.67%) nginx
7 1 (1.33%) Apache Airflow
7 1 (1.33%) Gunicorn
Cloud Services
1 34 (45.33%) Amazon AWS
2 13 (17.33%) Microsoft Azure
3 8 (10.67%) Azure Machine Learning
4 6 (8.00%) Amazon EC2
4 6 (8.00%) Amazon EMR
4 6 (8.00%) Amazon S3
5 4 (5.33%) Google Analytics
6 2 (2.67%) Google Cloud Platform
7 1 (1.33%) Amazon CloudWatch
7 1 (1.33%) Amazon Kinesis
7 1 (1.33%) AWS Lambda
7 1 (1.33%) Cloud Computing
7 1 (1.33%) IBM Cloud
7 1 (1.33%) SaaS
7 1 (1.33%) Serverless
7 1 (1.33%) Virtual Private Cloud
Communications & Networking
1 1 (1.33%) SSH
Database & Business Intelligence
1 30 (40.00%) Hadoop
2 25 (33.33%) Big Data
3 18 (24.00%) Apache Hive
4 13 (17.33%) Geospatial Data
4 13 (17.33%) MongoDB
5 12 (16.00%) HBase
6 8 (10.67%) Mahout
7 7 (9.33%) PostgreSQL
7 7 (9.33%) Power BI
8 6 (8.00%) NoSQL
9 4 (5.33%) Amazon Redshift
9 4 (5.33%) Data Warehouse
10 2 (2.67%) Amazon RDS
10 2 (2.67%) Apache Cassandra
10 2 (2.67%) Data Lake
10 2 (2.67%) Pentaho
11 1 (1.33%) Amazon DynamoDB
11 1 (1.33%) Redis
11 1 (1.33%) Riak
11 1 (1.33%) Sqoop
Development Applications
1 2 (2.67%) Git (software)
2 1 (1.33%) Jenkins
2 1 (1.33%) JIRA
2 1 (1.33%) Mercurial
2 1 (1.33%) Subversion
2 1 (1.33%) TeamCity
General
1 11 (14.67%) Finance
2 5 (6.67%) Retail
3 4 (5.33%) Advertising
3 4 (5.33%) Banking
4 3 (4.00%) Games
5 1 (1.33%) Aerospace
5 1 (1.33%) Marketing
Job Titles
1 30 (40.00%) Data Scientist
2 22 (29.33%) Developer
3 19 (25.33%) Python Developer
4 13 (17.33%) Data Engineer
5 9 (12.00%) AWS Engineer
6 8 (10.67%) Machine Learning Scientist
7 6 (8.00%) Python Engineer
7 6 (8.00%) Senior Data Engineer
8 4 (5.33%) Big Data Engineer
8 4 (5.33%) Machine Learning Engineer
9 3 (4.00%) Games Engineer
10 2 (2.67%) Data Science Leader
10 2 (2.67%) Senior Data Scientist
11 1 (1.33%) Cloud Engineer
11 1 (1.33%) Data Architect
11 1 (1.33%) Hadoop Engineer
11 1 (1.33%) Linux Developer
11 1 (1.33%) React Developer
11 1 (1.33%) Senior Python Developer
11 1 (1.33%) SQL Developer
Libraries, Frameworks & Software Standards
1 22 (29.33%) Pandas
2 14 (18.67%) scikit-learn
3 13 (17.33%) TensorFlow
3 13 (17.33%) Web Services
4 11 (14.67%) Keras
5 10 (13.33%) REST
6 9 (12.00%) NumPy
7 8 (10.67%) Matplotlib
7 8 (10.67%) Seaborn
8 6 (8.00%) Kafka
9 1 (1.33%) GraphX
9 1 (1.33%) Java EE
9 1 (1.33%) jQuery
9 1 (1.33%) MLlib
9 1 (1.33%) Natural Language Processing
9 1 (1.33%) RabbitMQ
9 1 (1.33%) React
9 1 (1.33%) SciPy
9 1 (1.33%) SparkR
9 1 (1.33%) Theano
Miscellaneous
1 16 (21.33%) Computer Science
2 9 (12.00%) Algorithms
3 4 (5.33%) Data Structures
3 4 (5.33%) Fintech
4 1 (1.33%) Fixed Income
4 1 (1.33%) GPGPU
4 1 (1.33%) Greenfield Project
4 1 (1.33%) Social Media
4 1 (1.33%) User Experience
Operating Systems
1 14 (18.67%) Linux
2 1 (1.33%) Ubuntu
2 1 (1.33%) Unix
2 1 (1.33%) Windows
Processes & Methodologies
1 48 (64.00%) Data Science
2 45 (60.00%) Machine Learning
3 27 (36.00%) Analytics
4 19 (25.33%) Deep Learning
5 17 (22.67%) Mathematics
6 16 (21.33%) Artificial Intelligence
7 15 (20.00%) Agile Software Development
8 14 (18.67%) TDD
9 13 (17.33%) Continuous Integration
9 13 (17.33%) Visualisation
10 12 (16.00%) Physics
11 11 (14.67%) Business Intelligence
11 11 (14.67%) Deployment Automation
11 11 (14.67%) MapReduce
12 10 (13.33%) Data Engineering
12 10 (13.33%) SOA
13 9 (12.00%) ETL
14 8 (10.67%) Logistic Regression
14 8 (10.67%) Statistics
14 8 (10.67%) Trend Analysis
Programming Languages
1 66 (88.00%) Python
2 26 (34.67%) SQL
3 19 (25.33%) Scala
4 6 (8.00%) Java
5 3 (4.00%) R
6 2 (2.67%) Shell Script
7 1 (1.33%) Bash Shell
7 1 (1.33%) C++
Qualifications
1 12 (16.00%) Degree
2 8 (10.67%) BPSS Clearance
3 2 (2.67%) PhD
Quality Assurance & Compliance
1 1 (1.33%) Data Quality
1 1 (1.33%) GCP
System Software
1 5 (6.67%) Apache Flume
1 5 (6.67%) Docker
2 2 (2.67%) HDFS
Systems Management
1 5 (6.67%) Oozie
2 2 (2.67%) Ansible
2 2 (2.67%) Apache Slider
2 2 (2.67%) Hadoop YARN
3 1 (1.33%) Ambari
3 1 (1.33%) Packer
3 1 (1.33%) Terraform
Vendors
1 8 (10.67%) MapR
1 8 (10.67%) Tableau
2 6 (8.00%) Google
3 4 (5.33%) Salesforce.com
4 2 (2.67%) Attivio
5 1 (1.33%) Cloudera
5 1 (1.33%) Hortonworks
5 1 (1.33%) IBM
5 1 (1.33%) Microsoft
5 1 (1.33%) Vertica