Period
to 21 November 2017

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 21 November 2017 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
21 Nov 2017
Same period 2016 Same period 2015
Rank 181 238 474
Rank change year-on-year +57 +236 +292
Contract jobs citing Apache Spark 1497 1160 467
As % of all contract IT jobs advertised in the UK 1.45% 1.11% 0.41%
As % of the Application Platforms category 8.73% 5.98% 2.33%
Number of daily rates quoted 1092 795 340
UK median daily rate £550 £550 £600
Median daily rate % change year-on-year - -8.33% +9.09%
10th Percentile £400 £400 £425
90th Percentile £675 £688 £700
UK excluding London median daily rate £475 £450 £513
% change year-on-year +5.56% -12.28% -10.78%
Number of hourly rates quoted 10 3 3
UK median hourly rate £23.42 £20.50 £33.55
Median hourly rate % change year-on-year +14.24% -38.90% -8.08%
UK excluding London median hourly rate £23.42 £20.50 £33.55
% change year-on-year +14.24% -38.90% -8.08%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 17155 19384 20012
As % of all contract IT jobs advertised in the UK 16.64% 18.51% 17.62%
Number of daily rates quoted 11290 12557 12574
UK median daily rate £450 £420 £400
Median daily rate % change year-on-year +7.14% +5.00% +6.45%
10th Percentile £263 £250 £230
90th Percentile £605 £588 £563
UK excluding London median daily rate £400 £388 £375
% change year-on-year +3.23% +3.33% +3.45%
Number of hourly rates quoted 434 478 605
UK median hourly rate £20.00 £20.00 £20.00
10th Percentile £11.00 £12.25 £11.00
90th Percentile £47.83 £48.75 £42.75
UK excluding London median hourly rate £20.00 £20.73 £20.00
% change year-on-year -3.52% +3.65% +14.29%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 21 November 2017.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 21 November 2017.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 21 November 2017. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +60 1462 £550 - 216
London +14 1012 £550 - 166
UK excluding London +114 471 £475 +5.56% 53
South East +93 234 £500 -4.76% 14
North of England +95 87 £460 -8.00% 20
East of England +27 61 £488 -7.14% 9
Yorkshire +41 49 £438 -12.06% 4
Midlands +33 43 £450 +5.88% 1
North West +50 39 £506 -3.57% 16
West Midlands +51 37 £450 +9.09% 1
Scotland +4 22 £400 -11.11% 2
South West -44 20 £438 +19.05% 6
North East +70 7 £400 -20.00%
East Midlands -16 6 £493 +15.88%
Wales -3 2 - -
Northern Ireland -46 2 - - 1

For the 6 months to 21 November 2017, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the number co-occurrences and its proportion to all contract ads with a requirement for Apache Spark.

1 1158 (77.35%) Big Data
2 1122 (74.95%) Hadoop
3 807 (53.91%) Java
4 645 (43.09%) Python
5 621 (41.48%) Kafka
6 571 (38.14%) Agile Software Development
7 568 (37.94%) Scala
8 523 (34.94%) Apache Hive
9 441 (29.46%) SQL
10 394 (26.32%) HBase
11 366 (24.45%) Analytics
12 359 (23.98%) HDFS
13 344 (22.98%) TDD
14 313 (20.91%) Amazon AWS
15 305 (20.37%) Finance
16 299 (19.97%) Linux
17 292 (19.51%) NoSQL
18 275 (18.37%) Machine Learning
19 262 (17.50%) MapReduce
20 254 (16.97%) Scrum
21 249 (16.63%) Apache
22 246 (16.43%) DevOps
23 232 (15.50%) Apache Cassandra
24 228 (15.23%) Docker
25 217 (14.50%) Banking
26 216 (14.43%) ETL
27 214 (14.30%) Apache Flume
27 214 (14.30%) Cloudera
28 206 (13.76%) Spring
29 196 (13.09%) Impala

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 249 (16.63%) Apache
2 119 (7.95%) Elasticsearch
3 118 (7.88%) Apache Solr
4 93 (6.21%) Apache Pig
5 55 (3.67%) Confluence
6 48 (3.21%) SAS
7 34 (2.27%) Tomcat
8 22 (1.47%) Apache Airflow
8 22 (1.47%) Cloud Foundry
9 20 (1.34%) JBoss
10 16 (1.07%) GemFire
10 16 (1.07%) OpenStack
11 13 (0.87%) Oracle SOA Suite
12 12 (0.80%) Jupyter
13 11 (0.73%) GlassFish
13 11 (0.73%) WildFly
14 10 (0.67%) Luigi
15 8 (0.53%) Oracle Coherence
16 6 (0.40%) nginx
17 4 (0.27%) ArcGIS
Applications
1 16 (1.07%) SPSS
2 12 (0.80%) Chatbot
3 10 (0.67%) Pipeline Pilot
4 7 (0.47%) Microsoft Excel
5 6 (0.40%) Microsoft Office
6 5 (0.33%) Stata
7 3 (0.20%) Weka
8 1 (0.067%) Adobe Illustrator
8 1 (0.067%) GNU Octave
8 1 (0.067%) Microsoft PowerPoint
8 1 (0.067%) MS Visio
8 1 (0.067%) Photoshop
Business Applications
1 2 (0.13%) SAP CRM
2 1 (0.067%) Dynamics CRM
2 1 (0.067%) Oracle BPM
Cloud Services
1 313 (20.91%) Amazon AWS
2 125 (8.35%) GitHub
3 75 (5.01%) Microsoft Azure
4 70 (4.68%) Google Cloud Platform
5 37 (2.47%) Amazon EMR
5 37 (2.47%) Bluemix
5 37 (2.47%) IBM Cloud
6 36 (2.40%) Amazon S3
7 29 (1.94%) PaaS
8 27 (1.80%) Amazon EC2
9 23 (1.54%) SaaS
10 20 (1.34%) Amazon Kinesis
11 12 (0.80%) Cloud Computing
11 12 (0.80%) Google Analytics
12 11 (0.73%) Virtual Private Cloud
13 10 (0.67%) Azure Data Factory
13 10 (0.67%) Dialogflow
13 10 (0.67%) OpenShift
14 8 (0.53%) Azure Machine Learning
14 8 (0.53%) IaaS
Communications & Networking
1 126 (8.42%) Kerberos
2 27 (1.80%) HTTP
3 21 (1.40%) Firewall
4 20 (1.34%) DNS
5 19 (1.27%) VPN
6 15 (1.00%) LAN
7 14 (0.94%) WAN
8 5 (0.33%) ICMP
8 5 (0.33%) TCP/IP
9 4 (0.27%) Internet
9 4 (0.27%) LDAP
10 3 (0.20%) Intranet
11 2 (0.13%) Network Security
11 2 (0.13%) SNA
12 1 (0.067%) Broadband
12 1 (0.067%) Multicast
12 1 (0.067%) OpenLDAP
12 1 (0.067%) SSL
Database & Business Intelligence
1 1158 (77.35%) Big Data
2 1122 (74.95%) Hadoop
3 523 (34.94%) Apache Hive
4 394 (26.32%) HBase
5 292 (19.51%) NoSQL
6 232 (15.50%) Apache Cassandra
7 196 (13.09%) Impala
8 170 (11.36%) Sqoop
9 169 (11.29%) MongoDB
10 149 (9.95%) PostgreSQL
11 138 (9.22%) Data Warehouse
12 109 (7.28%) Relational Database
13 86 (5.74%) SQL Server
14 61 (4.07%) Amazon Redshift
15 51 (3.41%) MySQL
16 49 (3.27%) RDBMS
17 40 (2.67%) Flink
18 39 (2.61%) Data Lake
19 35 (2.34%) Data Mining
20 32 (2.14%) Spark SQL
Development Applications
1 186 (12.42%) Git (software)
2 178 (11.89%) Jenkins
3 122 (8.15%) JIRA
4 87 (5.81%) Cucumber
5 46 (3.07%) JUnit
6 43 (2.87%) Maven
7 33 (2.20%) Hudson
8 29 (1.94%) Talend Open Studio
9 28 (1.87%) Gradle
10 26 (1.74%) IntelliJ
11 21 (1.40%) Mockito
12 20 (1.34%) Subversion
13 18 (1.20%) TeamCity
14 17 (1.14%) Team Foundation Server
15 16 (1.07%) GitLab
16 15 (1.00%) Sonatype Nexus
17 11 (0.73%) JBehave
18 10 (0.67%) Visual Studio
19 9 (0.60%) Eclipse
19 9 (0.60%) Selenium
General
1 305 (20.37%) Finance
2 217 (14.50%) Banking
3 85 (5.68%) Investment Banking
4 81 (5.41%) Telecoms
5 72 (4.81%) Marketing
5 72 (4.81%) Retail
6 50 (3.34%) Front Office
7 45 (3.01%) Electronics
8 44 (2.94%) Manufacturing
9 36 (2.40%) Publishing
10 16 (1.07%) Games
11 10 (0.67%) Pharmaceutical
12 6 (0.40%) Advertising
12 6 (0.40%) Retail Banking
13 4 (0.27%) Automotive
13 4 (0.27%) Law
14 3 (0.20%) Financial Institution
14 3 (0.20%) Legal
14 3 (0.20%) Spanish Language
15 2 (0.13%) Private Banking
Job Titles
1 560 (37.41%) Developer
2 390 (26.05%) Data Engineer
3 295 (19.71%) Big Data Engineer
4 246 (16.43%) Big Data Developer
5 198 (13.23%) Java Developer
6 158 (10.55%) Scala Developer
7 156 (10.42%) Architect
8 140 (9.35%) Data Scientist
9 135 (9.02%) Hadoop Engineer
10 112 (7.48%) Apache Spark Developer
11 109 (7.28%) Data Architect
12 107 (7.15%) Hadoop Developer
13 83 (5.54%) Java Engineer
14 82 (5.48%) Big Data Architect
15 57 (3.81%) DevOps Engineer
16 53 (3.54%) Senior Developer
17 50 (3.34%) AWS Engineer
18 38 (2.54%) Analyst
19 37 (2.47%) Java Hadoop Developer
19 37 (2.47%) Senior Data Engineer
Libraries, Frameworks & Software Standards
1 621 (41.48%) Kafka
2 206 (13.76%) Spring
3 172 (11.49%) REST
4 143 (9.55%) Spring Boot
5 132 (8.82%) RESTful
6 120 (8.02%) Spark Streaming
7 97 (6.48%) JSON
8 94 (6.28%) PySpark
9 87 (5.81%) Web Services
10 84 (5.61%) Akka
11 80 (5.34%) Drools
12 59 (3.94%) Apache Camel
13 56 (3.74%) XML
14 54 (3.61%) React
15 52 (3.47%) AngularJS
16 50 (3.34%) Play Framework
17 46 (3.07%) Node.js
18 44 (2.94%) Elastic Stack
19 43 (2.87%) Natural Language Processing
20 42 (2.81%) Hibernate
Miscellaneous
1 187 (12.49%) Algorithms
2 159 (10.62%) Analytical Skills
3 66 (4.41%) Computer Science
4 56 (3.74%) Greenfield Project
5 36 (2.40%) Fintech
6 27 (1.80%) Market Risk
6 27 (1.80%) Replication
7 26 (1.74%) Derivative
8 24 (1.60%) User Experience
9 21 (1.40%) Data Structures
10 18 (1.20%) Foreign Exchange (FX)
10 18 (1.20%) Linux Command Line
11 16 (1.07%) Virtual Team
12 14 (0.94%) Life Science
12 14 (0.94%) Semantic Web
13 13 (0.87%) Mobile App
14 12 (0.80%) Credit Risk
14 12 (0.80%) Equities
14 12 (0.80%) Management Information System
15 10 (0.67%) Data Centre
Operating Systems
1 299 (19.97%) Linux
2 156 (10.42%) Unix
3 44 (2.94%) Red Hat Enterprise Linux
4 23 (1.54%) CentOS
5 21 (1.40%) Ubuntu
6 16 (1.07%) Android
6 16 (1.07%) Apple iOS
7 14 (0.94%) Windows
8 1 (0.067%) CoreOS
8 1 (0.067%) Debian
8 1 (0.067%) Mac OS X
8 1 (0.067%) VMS
8 1 (0.067%) Windows 10
8 1 (0.067%) Windows Server
8 1 (0.067%) Windows Server 2008
Processes & Methodologies
1 571 (38.14%) Agile Software Development
2 366 (24.45%) Analytics
3 344 (22.98%) TDD
4 275 (18.37%) Machine Learning
5 262 (17.50%) MapReduce
6 254 (16.97%) Scrum
7 246 (16.43%) DevOps
8 216 (14.43%) ETL
9 192 (12.83%) Open Source
10 187 (12.49%) BDD
11 182 (12.16%) Microservices
12 164 (10.96%) Business Intelligence
12 164 (10.96%) Continuous Integration
13 144 (9.62%) Software Engineering
14 143 (9.55%) Data Modelling
15 139 (9.29%) Data Architecture
16 106 (7.08%) Data Analytics
17 105 (7.01%) Visualisation
18 104 (6.95%) Data Engineering
19 88 (5.88%) Kanban
Programming Languages
1 807 (53.91%) Java
2 645 (43.09%) Python
3 568 (37.94%) Scala
4 441 (29.46%) SQL
5 182 (12.16%) R
6 112 (7.48%) Shell Script
7 96 (6.41%) JavaScript
8 55 (3.67%) C#
9 44 (2.94%) Bash Shell
10 39 (2.61%) Perl
11 33 (2.20%) Ruby
12 25 (1.67%) PL/SQL
13 19 (1.27%) C++
13 19 (1.27%) MATLAB
14 18 (1.20%) Haskell
14 18 (1.20%) SPARQL
15 13 (0.87%) C
16 12 (0.80%) PHP
17 11 (0.73%) Go
17 11 (0.73%) T-SQL
Qualifications
1 129 (8.62%) Security Cleared
2 112 (7.48%) SC Cleared
3 89 (5.95%) Degree
4 37 (2.47%) PhD
5 19 (1.27%) Computer Science Degree
6 13 (0.87%) CISSP
7 12 (0.80%) CEH
7 12 (0.80%) SANS
8 8 (0.53%) Numerate Degree
9 6 (0.40%) DBS Check
9 6 (0.40%) DV Cleared
9 6 (0.40%) PMP
10 5 (0.33%) ISTQB
11 4 (0.27%) Master's Degree
11 4 (0.27%) Postgraduate
12 3 (0.20%) CTC Cleared
12 3 (0.20%) ISTQB Foundation Certification
13 2 (0.13%) BPSS Clearance
13 2 (0.13%) Microsoft Certification
13 2 (0.13%) PMI Certification
Quality Assurance & Compliance
1 70 (4.68%) QA
2 29 (1.94%) Data Quality
3 14 (0.94%) GDPR
4 6 (0.40%) GCP
5 5 (0.33%) Actionable Recommendations
6 4 (0.27%) MiFID
7 3 (0.20%) IFRS
7 3 (0.20%) IFRS 9
8 1 (0.067%) Basel III
8 1 (0.067%) Dodd-Frank
8 1 (0.067%) FATCA
8 1 (0.067%) FedRAMP
8 1 (0.067%) PCI DSS
System Software
1 359 (23.98%) HDFS
2 228 (15.23%) Docker
3 214 (14.30%) Apache Flume
4 111 (7.41%) Apache ZooKeeper
5 12 (0.80%) Active Directory
6 10 (0.67%) Virtual Machines
7 6 (0.40%) Virtual Servers
8 2 (0.13%) VirtualBox
9 1 (0.067%) VMware Infrastructure
Systems Management
1 156 (10.42%) Oozie
2 142 (9.49%) Hadoop YARN
3 120 (8.02%) Ansible
4 101 (6.75%) Kubernetes
5 96 (6.41%) Terraform
6 92 (6.15%) Ambari
7 65 (4.34%) Puppet
8 40 (2.67%) Mesos
9 39 (2.61%) Opscode Chef
10 18 (1.20%) Kibana
11 17 (1.14%) Prometheus
12 16 (1.07%) ITRS Geneos
13 14 (0.94%) logstash
14 13 (0.87%) Control-M
15 5 (0.33%) yum
16 3 (0.20%) Apache Slider
16 3 (0.20%) Rundeck
17 2 (0.13%) Computer Emergency Response Teams
17 2 (0.13%) Marathon
17 2 (0.13%) Packer
Vendors
1 214 (14.30%) Cloudera
2 133 (8.88%) Oracle
3 115 (7.68%) Tableau
4 109 (7.28%) Hortonworks
5 101 (6.75%) Google
6 67 (4.48%) IBM
7 59 (3.94%) SAP
8 50 (3.34%) Microsoft
9 47 (3.14%) Red Hat
10 36 (2.40%) Splunk
11 29 (1.94%) Talend
12 27 (1.80%) Informatica
13 26 (1.74%) MapR
14 24 (1.60%) Atlassian
15 21 (1.40%) Salesforce.com
16 16 (1.07%) AppDynamics
16 16 (1.07%) Teradata
17 10 (0.67%) Netezza
17 10 (0.67%) RapidMiner
18 8 (0.53%) Sybase