Period
to 16 July 2018

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 16 July 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
16 Jul 2018
Same period 2017 Same period 2016
Rank 178 212 382
Rank change year-on-year +34 +170 +272
Contract jobs citing Apache Spark 1,580 1,188 688
As % of all contract IT jobs advertised in the UK 1.52% 1.25% 0.62%
As % of the Application Platforms category 9.17% 7.37% 3.51%
Number of daily rates quoted 1,121 820 476
UK median daily rate £550 £550 £550
10th Percentile £416 £400 £411
90th Percentile £688 £688 £700
UK excluding London median daily rate £500 £475 £425
% change year-on-year +5.26% +11.76% -5.56%
Number of hourly rates quoted 8 1 0
UK median hourly rate £35.00 £35.00 -
UK excluding London median hourly rate £22.25 £35.00 -
% change year-on-year -36.43% - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 17,222 16,110 19,601
As % of all contract IT jobs advertised in the UK 16.60% 16.97% 17.80%
Number of daily rates quoted 11,309 10,400 12,823
UK median daily rate £450 £430 £418
Median daily rate % change year-on-year +4.65% +2.99% +7.05%
10th Percentile £263 £263 £250
90th Percentile £625 £600 £588
UK excluding London median daily rate £415 £400 £393
% change year-on-year +3.75% +1.91% +4.67%
Number of hourly rates quoted 402 394 472
UK median hourly rate £20.00 £19.64 £20.00
Median hourly rate % change year-on-year +1.82% -1.79% +5.26%
10th Percentile £12.00 £10.75 £11.25
90th Percentile £51.13 £52.50 £45.73
UK excluding London median hourly rate £19.00 £19.50 £20.39
% change year-on-year -2.56% -4.36% +10.22%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 16 July 2018.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 16 July 2018.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 16 July 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +31 1,507 £550 - 263
London -4 1,007 £558 +1.36% 174
UK excluding London +101 559 £500 +5.26% 110
South East +60 236 £500 +5.26% 31
North of England +42 128 £500 +6.38% 26
Yorkshire +17 73 £475 +2.48% 7
East of England +7 52 £475 -1.30% 7
North West +6 51 £550 +7.32% 17
South West +30 40 £488 +21.88% 16
Midlands -8 40 £500 +5.26% 9
East Midlands +24 32 £500 -4.76% 7
Scotland -17 31 £475 - 16
Northern Ireland +36 20 £432 - 4
Wales -16 12 £550 +10.00% 1
West Midlands -32 8 £400 -15.79% 2
North East +1 4 £588 +38.24% 2

For the 6 months to 16 July 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 1,141 (72.22%) Big Data
2 1,136 (71.90%) Hadoop
3 768 (48.61%) Python
4 754 (47.72%) Kafka
5 735 (46.52%) Java
6 619 (39.18%) Agile Software Development
7 615 (38.92%) Scala
8 569 (36.01%) Apache Hive
9 560 (35.44%) SQL
10 492 (31.14%) Analytics
11 435 (27.53%) Amazon AWS
12 431 (27.28%) Finance
13 387 (24.49%) Data Science
14 384 (24.30%) NoSQL
15 374 (23.67%) HBase
16 353 (22.34%) Apache
17 349 (22.09%) Linux
18 329 (20.82%) Machine Learning
19 304 (19.24%) DevOps
20 280 (17.72%) ETL
21 277 (17.53%) HDFS
22 272 (17.22%) Data Warehouse
23 242 (15.32%) TDD
24 237 (15.00%) Apache Cassandra
25 236 (14.94%) Git (software)
26 226 (14.30%) Jenkins
27 214 (13.54%) Microsoft Azure
28 213 (13.48%) Docker
29 211 (13.35%) Oozie
29 211 (13.35%) MongoDB

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 353 (22.34%) Apache
2 144 (9.11%) Apache Pig
3 108 (6.84%) Elasticsearch
4 94 (5.95%) Apache Solr
5 63 (3.99%) SAS
6 47 (2.97%) Apache Airflow
7 39 (2.47%) Cloud Foundry
8 33 (2.09%) Confluence
9 28 (1.77%) nginx
10 18 (1.14%) Tomcat
11 17 (1.08%) SharePoint
12 16 (1.01%) JBoss
13 14 (0.89%) Luigi
14 13 (0.82%) GemFire
15 11 (0.70%) Gunicorn
16 8 (0.51%) Oracle Workflow
17 7 (0.44%) Jupyter
18 4 (0.25%) Oracle SOA Suite
19 3 (0.19%) Oracle Self-Service HR
19 3 (0.19%) WebLogic
Applications
1 11 (0.70%) Chatbot
1 11 (0.70%) Microsoft Excel
2 3 (0.19%) MS Visio
3 2 (0.13%) Microsoft Project
3 2 (0.13%) Shockwave
3 2 (0.13%) SPSS
4 1 (0.063%) InVision
4 1 (0.063%) Microsoft Office
4 1 (0.063%) Microsoft PowerPoint
4 1 (0.063%) Photoshop
4 1 (0.063%) Stata
Business Applications
1 14 (0.89%) SAP Oil and Gas
2 3 (0.19%) Dynamics AX
2 3 (0.19%) Dynamics CRM
2 3 (0.19%) Oracle Financials
3 1 (0.063%) Finacle
3 1 (0.063%) FLEXCUBE
3 1 (0.063%) ION MarketView
3 1 (0.063%) Markit EDM
3 1 (0.063%) PolarLake
3 1 (0.063%) Remedy ITSM
Cloud Services
1 435 (27.53%) Amazon AWS
2 214 (13.54%) Microsoft Azure
3 93 (5.89%) Google Cloud Platform
4 91 (5.76%) Amazon EMR
5 68 (4.30%) Amazon S3
6 65 (4.11%) Amazon Kinesis
7 63 (3.99%) GitHub
8 52 (3.29%) Serverless
9 41 (2.59%) Amazon EC2
10 39 (2.47%) AWS Lambda
11 28 (1.77%) Azure Data Factory
12 27 (1.71%) PaaS
13 23 (1.46%) Cloud Computing
14 16 (1.01%) Data Lake Analytics
14 16 (1.01%) Heroku
15 15 (0.95%) Google Analytics
15 15 (0.95%) SaaS
16 13 (0.82%) Azure Machine Learning
17 10 (0.63%) OpenShift
17 10 (0.63%) Route 53
Communications & Networking
1 83 (5.25%) Kerberos
2 50 (3.16%) HTTP
3 40 (2.53%) SSL
4 27 (1.71%) DNS
5 24 (1.52%) TCP/IP
6 21 (1.33%) VPN
7 18 (1.14%) FTP
8 8 (0.51%) ICMP
9 6 (0.38%) DSL
9 6 (0.38%) Firewall
9 6 (0.38%) Internet
9 6 (0.38%) Network Security
9 6 (0.38%) SNA
10 4 (0.25%) SSH
11 2 (0.13%) RTMP
12 1 (0.063%) AMQP
12 1 (0.063%) BIG-IP
12 1 (0.063%) Intranet
12 1 (0.063%) SAN
12 1 (0.063%) SMS
Database & Business Intelligence
1 1,141 (72.22%) Big Data
2 1,136 (71.90%) Hadoop
3 569 (36.01%) Apache Hive
4 384 (24.30%) NoSQL
5 374 (23.67%) HBase
6 272 (17.22%) Data Warehouse
7 237 (15.00%) Apache Cassandra
8 211 (13.35%) MongoDB
9 193 (12.22%) Relational Database
10 192 (12.15%) SQL Server
11 163 (10.32%) PostgreSQL
12 159 (10.06%) Impala
13 136 (8.61%) Data Lake
14 133 (8.42%) Amazon Redshift
15 109 (6.90%) Sqoop
16 101 (6.39%) RDBMS
17 99 (6.27%) MySQL
18 73 (4.62%) Power BI
19 64 (4.05%) Star Schema
20 63 (3.99%) OLAP
Development Applications
1 236 (14.94%) Git (software)
2 226 (14.30%) Jenkins
3 69 (4.37%) JIRA
4 47 (2.97%) GitLab
5 46 (2.91%) Cucumber
5 46 (2.91%) Visual Studio Team System
6 38 (2.41%) Maven
7 32 (2.03%) Subversion
8 25 (1.58%) Gradle
8 25 (1.58%) Selenium
9 24 (1.52%) Sonatype Nexus
10 22 (1.39%) IntelliJ
11 21 (1.33%) Atlassian Bamboo
11 21 (1.33%) JUnit
12 20 (1.27%) Team Foundation Server
13 18 (1.14%) TeamCity
14 17 (1.08%) Artifactory
15 16 (1.01%) Eclipse
16 14 (0.89%) gulp
16 14 (0.89%) SonarQube
General
1 431 (27.28%) Finance
2 163 (10.32%) Banking
3 64 (4.05%) Retail
4 47 (2.97%) Investment Banking
5 38 (2.41%) Games
6 36 (2.28%) Marketing
7 26 (1.65%) Front Office
7 26 (1.65%) Telecoms
8 20 (1.27%) Manufacturing
9 14 (0.89%) Advertising
9 14 (0.89%) Retail Banking
10 9 (0.57%) Legal
11 7 (0.44%) Publishing
12 6 (0.38%) Prime Brokerage
12 6 (0.38%) Private Banking
13 5 (0.32%) Electronics
13 5 (0.32%) Financial Institution
14 4 (0.25%) Law
15 3 (0.19%) Back Office
15 3 (0.19%) Digital Economy
Job Titles
1 587 (37.15%) Developer
2 456 (28.86%) Data Engineer
3 271 (17.15%) Big Data Engineer
4 259 (16.39%) Big Data Developer
5 172 (10.89%) Java Developer
6 155 (9.81%) Architect
7 133 (8.42%) Hadoop Engineer
8 118 (7.47%) Apache Spark Developer
9 111 (7.03%) Data Architect
10 105 (6.65%) Data Scientist
11 100 (6.33%) Hadoop Developer
12 96 (6.08%) Scala Developer
13 89 (5.63%) AWS Engineer
14 83 (5.25%) Big Data Architect
15 73 (4.62%) Python Developer
16 72 (4.56%) Senior Data Engineer
17 71 (4.49%) DevOps Engineer
18 61 (3.86%) Python Engineer
19 57 (3.61%) Consultant
20 54 (3.42%) Senior Developer
Libraries, Frameworks & Software Standards
1 754 (47.72%) Kafka
2 160 (10.13%) Spark Streaming
3 132 (8.35%) RabbitMQ
4 127 (8.04%) Spring
5 125 (7.91%) JSON
6 124 (7.85%) REST
7 123 (7.78%) RESTful
8 83 (5.25%) AngularJS
9 76 (4.81%) J2EE
10 74 (4.68%) Spring Boot
11 66 (4.18%) Akka
12 64 (4.05%) HTML
13 63 (3.99%) Middleware
14 61 (3.86%) Node.js
15 60 (3.80%) React
15 60 (3.80%) Web Services
16 58 (3.67%) XML
17 55 (3.48%) TensorFlow
18 50 (3.16%) Natural Language Processing
19 46 (2.91%) HTML5
Miscellaneous
1 164 (10.38%) Algorithms
2 148 (9.37%) Analytical Skills
3 117 (7.41%) Computer Science
4 105 (6.65%) Greenfield Project
5 62 (3.92%) Fintech
6 43 (2.72%) Data Structures
7 30 (1.90%) Linux Command Line
8 28 (1.77%) Derivative
8 28 (1.77%) Self-Motivation
9 24 (1.52%) Enterprise Software
10 23 (1.46%) HoloLens
10 23 (1.46%) User Experience
11 22 (1.39%) Virtual Team
12 16 (1.01%) Credit Risk
13 13 (0.82%) Fixed Income
14 11 (0.70%) Management Information System
15 10 (0.63%) NHS
15 10 (0.63%) Public Cloud
16 9 (0.57%) Grid Computing
17 8 (0.51%) Market Risk
Operating Systems
1 349 (22.09%) Linux
2 162 (10.25%) Unix
3 32 (2.03%) Red Hat Enterprise Linux
4 25 (1.58%) Windows
5 17 (1.08%) CentOS
5 17 (1.08%) Ubuntu
6 7 (0.44%) DC/OS
7 5 (0.32%) Windows Server
7 5 (0.32%) Windows Server 2008
8 4 (0.25%) Android
8 4 (0.25%) Apple iOS
8 4 (0.25%) Mac OS X
9 3 (0.19%) Debian
10 2 (0.13%) SUSE Linux Enterprise Server
Processes & Methodologies
1 619 (39.18%) Agile Software Development
2 492 (31.14%) Analytics
3 387 (24.49%) Data Science
4 329 (20.82%) Machine Learning
5 304 (19.24%) DevOps
6 280 (17.72%) ETL
7 242 (15.32%) TDD
8 200 (12.66%) Data Modelling
9 199 (12.59%) Scrum
10 190 (12.03%) Continuous Integration
11 187 (11.84%) Business Intelligence
12 161 (10.19%) Data Engineering
13 159 (10.06%) Open Source
14 145 (9.18%) Microservices
15 142 (8.99%) Visualisation
16 130 (8.23%) MapReduce
17 126 (7.97%) BDD
18 124 (7.85%) Software Engineering
19 122 (7.72%) Test Automation
20 121 (7.66%) Problem-Solving
Programming Languages
1 768 (48.61%) Python
2 735 (46.52%) Java
3 615 (38.92%) Scala
4 560 (35.44%) SQL
5 161 (10.19%) R
6 131 (8.29%) JavaScript
7 98 (6.20%) Java 8
8 85 (5.38%) Bash Shell
9 84 (5.32%) Shell Script
10 63 (3.99%) C++
11 49 (3.10%) Perl
11 49 (3.10%) Ruby
12 43 (2.72%) C#
13 32 (2.03%) C
14 30 (1.90%) MATLAB
15 23 (1.46%) PL/SQL
15 23 (1.46%) T-SQL
16 21 (1.33%) TypeScript
17 20 (1.27%) ES6
17 20 (1.27%) Groovy
Qualifications
1 131 (8.29%) Degree
2 86 (5.44%) Security Cleared
3 68 (4.30%) SC Cleared
4 46 (2.91%) Computer Science Degree
5 26 (1.65%) PhD
6 20 (1.27%) Master's Degree
7 14 (0.89%) DV Cleared
8 12 (0.76%) ISTQB
9 11 (0.70%) Numerate Degree
10 8 (0.51%) Microsoft Certification
11 3 (0.19%) AWS Certification
11 3 (0.19%) BPSS Clearance
11 3 (0.19%) CTC Cleared
12 2 (0.13%) CCA
12 2 (0.13%) ISEB
12 2 (0.13%) Sun Certification
13 1 (0.063%) CIPD
13 1 (0.063%) CISSP
13 1 (0.063%) MBA
13 1 (0.063%) MCSD
Quality Assurance & Compliance
1 105 (6.65%) Data Quality
2 53 (3.35%) QA
3 44 (2.78%) GCP
4 20 (1.27%) GDPR
5 7 (0.44%) PMO
6 4 (0.25%) SLA
7 2 (0.13%) ISO/IEC 27001
7 2 (0.13%) NIST 800
8 1 (0.063%) Disclosure Scotland
8 1 (0.063%) EMIR
8 1 (0.063%) MiFID
8 1 (0.063%) PSD2
System Software
1 277 (17.53%) HDFS
2 213 (13.48%) Docker
3 143 (9.05%) Apache Flume
4 119 (7.53%) Apache ZooKeeper
5 19 (1.20%) Active Directory
6 8 (0.51%) Virtual Servers
7 5 (0.32%) Virtual Machines
8 2 (0.13%) Ceph
9 1 (0.063%) etcd
9 1 (0.063%) Varnish
Systems Management
1 211 (13.35%) Oozie
2 174 (11.01%) Hadoop YARN
3 116 (7.34%) Ansible
4 105 (6.65%) Kubernetes
5 80 (5.06%) Terraform
6 68 (4.30%) Puppet
7 40 (2.53%) Opscode Chef
8 38 (2.41%) Mesos
9 34 (2.15%) Ambari
10 22 (1.39%) Kibana
11 16 (1.01%) Consul
12 14 (0.89%) Apache Slider
13 13 (0.82%) Marathon
13 13 (0.82%) Salt
14 11 (0.70%) Docker Swarm
14 11 (0.70%) Packer
14 11 (0.70%) Prometheus
15 7 (0.44%) ITRS Geneos
16 6 (0.38%) Host Intrusion Detection System
16 6 (0.38%) Sysdig
Vendors
1 198 (12.53%) Cloudera
2 169 (10.70%) Oracle
3 128 (8.10%) Hortonworks
4 99 (6.27%) Google
5 96 (6.08%) Microsoft
6 88 (5.57%) Talend
7 85 (5.38%) Tableau
8 68 (4.30%) Informatica
9 51 (3.23%) MapR
10 49 (3.10%) SAP
11 47 (2.97%) Splunk
12 46 (2.91%) Alteryx
13 33 (2.09%) IBM
14 29 (1.84%) Salesforce.com
15 24 (1.52%) Databricks
16 18 (1.14%) Atlassian
16 18 (1.14%) Sybase
17 15 (0.95%) SaltStack
17 15 (0.95%) Teradata
18 14 (0.89%) Red Hat