Period
to 18 February 2018

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 18 February 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
18 Feb 2018
Same period 2017 Same period 2016
Rank 175 197 444
Rank change year-on-year +22 +247 +237
Contract jobs citing Apache Spark 1,482 1,275 493
As % of all contract IT jobs advertised in the UK 1.48% 1.34% 0.46%
As % of the Application Platforms category 9.11% 7.51% 2.66%
Number of daily rates quoted 1,072 929 355
UK median daily rate £550 £550 £575
Median daily rate % change year-on-year - -4.35% -
10th Percentile £406 £413 £431
90th Percentile £675 £713 £675
UK excluding London median daily rate £500 £500 £550
% change year-on-year - -9.09% +83.33%
Number of hourly rates quoted 8 1 2
UK median hourly rate £29.00 £100.00 £35.53
Median hourly rate % change year-on-year -71.00% +181.49% -2.67%
10th Percentile £19.34 £90.00 £33.82
90th Percentile £60.63 £110.00 £38.23
UK excluding London median hourly rate £27.50 - £33.55

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 16,261 16,970 18,517
As % of all contract IT jobs advertised in the UK 16.28% 17.82% 17.15%
Number of daily rates quoted 10,862 10,931 11,752
UK median daily rate £450 £425 £400
Median daily rate % change year-on-year +5.88% +6.25% +1.27%
10th Percentile £263 £254 £235
90th Percentile £625 £600 £575
UK excluding London median daily rate £400 £400 £398
% change year-on-year - +0.63% +6.00%
Number of hourly rates quoted 384 360 507
UK median hourly rate £19.00 £19.00 £20.00
Median hourly rate % change year-on-year - -5.00% +11.11%
10th Percentile £11.25 £12.00 £10.98
90th Percentile £47.50 £47.50 £47.05
UK excluding London median hourly rate £18.05 £19.00 £19.00
% change year-on-year -4.99% - +8.82%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 18 February 2018.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 18 February 2018.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 18 February 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +26 1,453 £550 - 377
London -32 960 £575 - 273
UK excluding London +123 510 £500 - 117
South East +96 283 £500 -4.76% 30
North of England +61 78 £506 -0.25% 35
Midlands +26 52 £475 +5.56% 16
North West +35 44 £550 +4.27% 13
East of England +6 42 £525 - 13
South West +9 36 £475 +16.56% 10
Yorkshire +12 34 £438 -7.89% 21
East Midlands +6 31 £490 -9.89% 4
West Midlands +19 21 £475 +11.76% 12
Scotland -17 16 £425 -5.56% 8
North East +46 6 £400 -27.27% 1
Wales -32 2 - - 5
Northern Ireland -11 1 - -

For the 6 months to 18 February 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 1,086 (73.28%) Big Data
2 1,057 (71.32%) Hadoop
3 682 (46.02%) Python
4 679 (45.82%) Java
5 602 (40.62%) Scala
6 590 (39.81%) Kafka
7 527 (35.56%) Agile Software Development
8 515 (34.75%) Apache Hive
9 424 (28.61%) SQL
10 399 (26.92%) Analytics
11 348 (23.48%) Data Science
12 346 (23.35%) HBase
13 307 (20.72%) Amazon AWS
14 304 (20.51%) Linux
15 294 (19.84%) Machine Learning
16 292 (19.70%) HDFS
17 289 (19.50%) Finance
18 286 (19.30%) NoSQL
19 278 (18.76%) Apache
20 245 (16.53%) Cloudera
21 235 (15.86%) DevOps
22 226 (15.25%) ETL
23 216 (14.57%) TDD
24 209 (14.10%) Apache Flume
24 209 (14.10%) Banking
25 203 (13.70%) Apache Cassandra
26 200 (13.50%) Docker
27 193 (13.02%) Oozie
28 192 (12.96%) Jenkins
29 191 (12.89%) MapReduce

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 278 (18.76%) Apache
2 131 (8.84%) Apache Pig
3 66 (4.45%) Apache Solr
4 63 (4.25%) Elasticsearch
5 58 (3.91%) SAS
6 30 (2.02%) Jupyter
7 28 (1.89%) Tomcat
8 25 (1.69%) Cloud Foundry
9 19 (1.28%) Apache Airflow
10 16 (1.08%) nginx
11 15 (1.01%) Confluence
12 10 (0.67%) GemFire
12 10 (0.67%) Oracle Coherence
13 9 (0.61%) SharePoint
14 8 (0.54%) JBoss
15 7 (0.47%) Luigi
16 5 (0.34%) GeoServer
17 4 (0.27%) OpenStack
18 3 (0.20%) ArcGIS
18 3 (0.20%) Oracle SOA Suite
Applications
1 18 (1.21%) SPSS
2 7 (0.47%) Chatbot
2 7 (0.47%) Pipeline Pilot
3 6 (0.40%) Microsoft Excel
4 5 (0.34%) Microsoft Office
5 3 (0.20%) Stata
6 2 (0.13%) Weka
7 1 (0.067%) Adobe Illustrator
7 1 (0.067%) GNU Octave
7 1 (0.067%) Microsoft PowerPoint
7 1 (0.067%) MS Visio
7 1 (0.067%) Photoshop
Business Applications
1 2 (0.13%) Dynamics CRM
1 2 (0.13%) Oracle BPM
1 2 (0.13%) SAP CRM
Cloud Services
1 307 (20.72%) Amazon AWS
2 100 (6.75%) Microsoft Azure
3 84 (5.67%) GitHub
4 72 (4.86%) Google Cloud Platform
5 55 (3.71%) Amazon S3
6 40 (2.70%) Amazon EMR
7 37 (2.50%) Amazon Kinesis
8 27 (1.82%) Amazon EC2
8 27 (1.82%) PaaS
9 25 (1.69%) Serverless
10 24 (1.62%) AWS Lambda
11 23 (1.55%) Bluemix
11 23 (1.55%) IBM Cloud
12 22 (1.48%) SaaS
13 21 (1.42%) Google Tag Manager
14 15 (1.01%) IaaS
15 14 (0.94%) Azure Data Factory
15 14 (0.94%) Cloud Computing
16 12 (0.81%) OpenShift
17 10 (0.67%) Virtual Private Cloud
Communications & Networking
1 87 (5.87%) Kerberos
2 21 (1.42%) HTTP
3 19 (1.28%) SSL
4 16 (1.08%) DNS
5 14 (0.94%) Firewall
6 8 (0.54%) Internet
7 7 (0.47%) ICMP
7 7 (0.47%) TCP/IP
7 7 (0.47%) VPN
8 6 (0.40%) OpenLDAP
9 5 (0.34%) DSL
10 4 (0.27%) Network Security
11 2 (0.13%) AMQP
11 2 (0.13%) BIG-IP
11 2 (0.13%) SAN
12 1 (0.067%) Multicast
Database & Business Intelligence
1 1,086 (73.28%) Big Data
2 1,057 (71.32%) Hadoop
3 515 (34.75%) Apache Hive
4 346 (23.35%) HBase
5 286 (19.30%) NoSQL
6 203 (13.70%) Apache Cassandra
7 175 (11.81%) Data Warehouse
8 170 (11.47%) MongoDB
9 167 (11.27%) Sqoop
10 157 (10.59%) Impala
11 135 (9.11%) SQL Server
12 128 (8.64%) PostgreSQL
13 113 (7.62%) Relational Database
14 86 (5.80%) Data Lake
15 78 (5.26%) Amazon Redshift
16 63 (4.25%) MySQL
17 51 (3.44%) RDBMS
18 40 (2.70%) Flink
19 39 (2.63%) Mahout
20 37 (2.50%) Cloudera CDH
Development Applications
1 192 (12.96%) Jenkins
2 170 (11.47%) Git (software)
3 57 (3.85%) JIRA
4 37 (2.50%) JUnit
4 37 (2.50%) Maven
5 36 (2.43%) Cucumber
6 33 (2.23%) Gradle
7 25 (1.69%) Sonatype Nexus
8 24 (1.62%) TeamCity
9 22 (1.48%) Subversion
10 18 (1.21%) GitLab
11 17 (1.15%) Team Foundation Server
12 15 (1.01%) Selenium
13 14 (0.94%) Hudson
13 14 (0.94%) Visual Studio
14 13 (0.88%) Artifactory
15 12 (0.81%) IntelliJ
16 11 (0.74%) GoCD
16 11 (0.74%) gulp
16 11 (0.74%) Mockito
General
1 289 (19.50%) Finance
2 209 (14.10%) Banking
3 76 (5.13%) Retail
4 70 (4.72%) Investment Banking
5 53 (3.58%) Marketing
6 47 (3.17%) Telecoms
7 41 (2.77%) Manufacturing
8 40 (2.70%) Publishing
9 39 (2.63%) Front Office
10 26 (1.75%) Games
11 21 (1.42%) Electronics
12 15 (1.01%) Retail Banking
13 12 (0.81%) Advertising
13 12 (0.81%) Automotive
14 9 (0.61%) Pharmaceutical
15 5 (0.34%) Financial Institution
15 5 (0.34%) Law
16 4 (0.27%) French Language
16 4 (0.27%) Private Banking
17 2 (0.13%) Spanish Language
Job Titles
1 489 (33.00%) Developer
2 390 (26.32%) Data Engineer
3 275 (18.56%) Big Data Engineer
4 216 (14.57%) Big Data Developer
5 174 (11.74%) Architect
6 160 (10.80%) Data Scientist
6 160 (10.80%) Java Developer
7 126 (8.50%) Data Architect
8 123 (8.30%) Scala Developer
9 106 (7.15%) Hadoop Engineer
10 100 (6.75%) Apache Spark Developer
11 90 (6.07%) Big Data Architect
12 85 (5.74%) Hadoop Developer
13 71 (4.79%) DevOps Engineer
14 57 (3.85%) Consultant
15 50 (3.37%) AWS Engineer
16 48 (3.24%) Java Engineer
17 43 (2.90%) Analyst
18 39 (2.63%) Senior Developer
19 36 (2.43%) Python Developer
Libraries, Frameworks & Software Standards
1 590 (39.81%) Kafka
2 150 (10.12%) RESTful
3 127 (8.57%) Spark Streaming
4 121 (8.16%) Spring
5 108 (7.29%) PySpark
6 87 (5.87%) Spring Boot
7 85 (5.74%) REST
8 70 (4.72%) Web Services
9 65 (4.39%) JSON
10 61 (4.12%) TensorFlow
11 59 (3.98%) Akka
12 54 (3.64%) React
13 52 (3.51%) RabbitMQ
13 52 (3.51%) XML
14 51 (3.44%) AngularJS
15 47 (3.17%) Natural Language Processing
16 45 (3.04%) Node.js
17 42 (2.83%) scikit-learn
18 41 (2.77%) Elastic Stack
18 41 (2.77%) Pandas
Miscellaneous
1 153 (10.32%) Algorithms
2 139 (9.38%) Analytical Skills
3 95 (6.41%) Greenfield Project
4 94 (6.34%) Computer Science
5 39 (2.63%) Fintech
6 30 (2.02%) Data Structures
6 30 (2.02%) Derivative
7 23 (1.55%) Market Risk
7 23 (1.55%) Virtual Team
8 20 (1.35%) Replication
8 20 (1.35%) User Experience
9 15 (1.01%) Foreign Exchange (FX)
9 15 (1.01%) Linux Command Line
10 14 (0.94%) Data Centre
10 14 (0.94%) Management Information System
11 11 (0.74%) Life Science
12 10 (0.67%) Public Cloud
13 9 (0.61%) Equities
13 9 (0.61%) Linked Data
14 8 (0.54%) Fixed Income
Operating Systems
1 304 (20.51%) Linux
2 102 (6.88%) Unix
3 52 (3.51%) Red Hat Enterprise Linux
4 39 (2.63%) CentOS
5 15 (1.01%) Windows
6 12 (0.81%) Ubuntu
7 6 (0.40%) CoreOS
8 3 (0.20%) DC/OS
8 3 (0.20%) Debian
8 3 (0.20%) Windows Server
8 3 (0.20%) Windows Server 2008
9 2 (0.13%) Android
9 2 (0.13%) Apple iOS
10 1 (0.067%) VMS
10 1 (0.067%) Windows 10
Processes & Methodologies
1 527 (35.56%) Agile Software Development
2 399 (26.92%) Analytics
3 348 (23.48%) Data Science
4 294 (19.84%) Machine Learning
5 235 (15.86%) DevOps
6 226 (15.25%) ETL
7 216 (14.57%) TDD
8 191 (12.89%) MapReduce
9 169 (11.40%) Scrum
10 161 (10.86%) Open Source
11 154 (10.39%) Data Engineering
12 151 (10.19%) Business Intelligence
13 144 (9.72%) Continuous Integration
14 126 (8.50%) Visualisation
15 122 (8.23%) Data Analytics
15 122 (8.23%) Data Modelling
16 118 (7.96%) Software Engineering
17 115 (7.76%) Microservices
18 89 (6.01%) Data Architecture
19 86 (5.80%) Test Automation
Programming Languages
1 682 (46.02%) Python
2 679 (45.82%) Java
3 602 (40.62%) Scala
4 424 (28.61%) SQL
5 190 (12.82%) R
6 102 (6.88%) JavaScript
7 82 (5.53%) Shell Script
8 72 (4.86%) Bash Shell
9 43 (2.90%) C#
10 39 (2.63%) Ruby
11 38 (2.56%) Perl
12 27 (1.82%) C
13 25 (1.69%) PL/SQL
14 21 (1.42%) MATLAB
15 19 (1.28%) C++
15 19 (1.28%) Go
15 19 (1.28%) Haskell
16 13 (0.88%) Groovy
17 9 (0.61%) PowerShell
17 9 (0.61%) SPARQL
Qualifications
1 105 (7.09%) Degree
2 80 (5.40%) Security Cleared
3 65 (4.39%) SC Cleared
4 43 (2.90%) PhD
5 20 (1.35%) Master's Degree
6 17 (1.15%) Computer Science Degree
7 10 (0.67%) Numerate Degree
8 6 (0.40%) CEH
8 6 (0.40%) CISSP
8 6 (0.40%) DV Cleared
8 6 (0.40%) SANS
9 5 (0.34%) PMP
10 4 (0.27%) Microsoft Certification
11 3 (0.20%) CTC Cleared
11 3 (0.20%) Postgraduate
12 2 (0.13%) AWS Certification
12 2 (0.13%) DBS Check
12 2 (0.13%) ISTQB
12 2 (0.13%) PRINCE2 Certification
13 1 (0.067%) ISTQB Foundation Certification
Quality Assurance & Compliance
1 64 (4.32%) QA
2 42 (2.83%) Data Quality
3 19 (1.28%) GDPR
4 10 (0.67%) GCP
5 3 (0.20%) IFRS
5 3 (0.20%) IFRS 9
5 3 (0.20%) SLA
6 1 (0.067%) Basel II
6 1 (0.067%) FATCA
6 1 (0.067%) MiFID
6 1 (0.067%) PMO
6 1 (0.067%) PSD2
System Software
1 292 (19.70%) HDFS
2 209 (14.10%) Apache Flume
3 200 (13.50%) Docker
4 103 (6.95%) Apache ZooKeeper
5 21 (1.42%) Active Directory
6 10 (0.67%) Virtual Servers
7 8 (0.54%) Virtual Machines
8 5 (0.34%) Squid
9 2 (0.13%) Varnish
9 2 (0.13%) VirtualBox
10 1 (0.067%) VMware Infrastructure
Systems Management
1 193 (13.02%) Oozie
2 158 (10.66%) Hadoop YARN
3 82 (5.53%) Ansible
4 77 (5.20%) Ambari
5 74 (4.99%) Kubernetes
6 71 (4.79%) Puppet
7 51 (3.44%) Terraform
8 43 (2.90%) Mesos
9 41 (2.77%) Opscode Chef
10 18 (1.21%) Kibana
11 17 (1.15%) ITRS Geneos
12 15 (1.01%) Apache Slider
13 13 (0.88%) logstash
14 8 (0.54%) Consul
14 8 (0.54%) Packer
15 7 (0.47%) Nagios
15 7 (0.47%) Prometheus
16 6 (0.40%) Salt
16 6 (0.40%) Sysdig
16 6 (0.40%) ZABBIX
Vendors
1 245 (16.53%) Cloudera
2 168 (11.34%) Hortonworks
3 143 (9.65%) Oracle
4 109 (7.35%) Tableau
5 105 (7.09%) Google
6 77 (5.20%) MapR
6 77 (5.20%) Microsoft
7 60 (4.05%) SAP
8 56 (3.78%) Talend
9 41 (2.77%) Informatica
10 40 (2.70%) IBM
11 37 (2.50%) Salesforce.com
11 37 (2.50%) Splunk
12 22 (1.48%) Atlassian
12 22 (1.48%) Red Hat
13 21 (1.42%) Tealium
14 18 (1.21%) Alteryx
15 17 (1.15%) AppDynamics
15 17 (1.15%) Teradata
16 13 (0.88%) Ab Initio