Period
to 20 April 2018

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 20 April 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
20 Apr 2018
Same period 2017 Same period 2016
Rank 175 212 439
Rank change year-on-year +37 +227 +224
Contract jobs citing Apache Spark 1,577 1,146 539
As % of all contract IT jobs advertised in the UK 1.56% 1.24% 0.51%
As % of the Application Platforms category 9.46% 7.23% 2.94%
Number of daily rates quoted 1,106 820 393
UK median daily rate £550 £550 £575
Median daily rate % change year-on-year - -4.35% -
10th Percentile £415 £405 £431
90th Percentile £676 £700 £693
UK excluding London median daily rate £500 £500 £531
% change year-on-year - -5.88% +27.25%
Number of hourly rates quoted 4 1 0
UK median hourly rate £39.00 £100.00 -
Median hourly rate % change year-on-year -61.00% - -
UK excluding London median hourly rate £30.50 - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 16,674 15,852 18,326
As % of all contract IT jobs advertised in the UK 16.48% 17.10% 17.18%
Number of daily rates quoted 11,051 10,218 11,748
UK median daily rate £450 £425 £410
Median daily rate % change year-on-year +5.88% +3.66% +2.50%
10th Percentile £263 £255 £243
90th Percentile £625 £600 £588
UK excluding London median daily rate £413 £390 £390
% change year-on-year +5.91% -0.13% +4.00%
Number of hourly rates quoted 384 327 460
UK median hourly rate £18.95 £19.50 £20.25
Median hourly rate % change year-on-year -2.85% -3.70% +9.46%
10th Percentile £11.25 £11.55 £11.00
90th Percentile £47.50 £50.00 £47.50
UK excluding London median hourly rate £17.50 £19.00 £20.00
% change year-on-year -7.89% -5.00% +14.29%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 20 April 2018.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 20 April 2018.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 20 April 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +41 1,532 £550 - 285
London -10 1,009 £550 -4.35% 200
UK excluding London +153 556 £500 - 94
South East +110 269 £525 +10.53% 37
North of England +57 109 £500 -1.96% 13
Midlands +30 62 £475 +0.47% 11
Yorkshire +30 61 £469 +1.35% 2
North West +7 47 £550 +4.27% 10
East Midlands +33 46 £475 -14.80% 2
East of England +24 44 £475 -9.52% 13
South West +19 38 £481 +20.31% 11
West Midlands +9 16 £475 +5.56% 9
Scotland -31 13 £500 +5.26% 5
Wales -24 11 £550 +4.76% 2
Northern Ireland +6 10 £432 - 2
North East +33 5 £400 -27.27% 1

For the 6 months to 20 April 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 1,136 (72.04%) Big Data
2 1,127 (71.46%) Hadoop
3 766 (48.57%) Python
4 690 (43.75%) Kafka
5 675 (42.80%) Java
6 664 (42.11%) Scala
7 582 (36.91%) Agile Software Development
8 579 (36.72%) Apache Hive
9 507 (32.15%) SQL
10 479 (30.37%) Analytics
11 391 (24.79%) Amazon AWS
12 389 (24.67%) HBase
12 389 (24.67%) Data Science
13 353 (22.38%) Finance
14 352 (22.32%) Machine Learning
15 345 (21.88%) Linux
16 341 (21.62%) Apache
17 335 (21.24%) NoSQL
18 290 (18.39%) DevOps
19 288 (18.26%) ETL
20 287 (18.20%) Cloudera
21 272 (17.25%) HDFS
22 253 (16.04%) Data Warehouse
23 232 (14.71%) Apache Cassandra
24 227 (14.39%) Oozie
25 220 (13.95%) Jenkins
26 217 (13.76%) TDD
27 205 (13.00%) Apache Flume
28 202 (12.81%) Docker
29 201 (12.75%) Git (software)

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 341 (21.62%) Apache
2 154 (9.77%) Apache Pig
3 70 (4.44%) Elasticsearch
4 58 (3.68%) Apache Solr
5 56 (3.55%) SAS
6 39 (2.47%) Cloud Foundry
7 34 (2.16%) nginx
7 34 (2.16%) Tomcat
8 29 (1.84%) Jupyter
9 25 (1.59%) Apache Airflow
10 15 (0.95%) JBoss
11 13 (0.82%) Confluence
11 13 (0.82%) SharePoint
12 10 (0.63%) GemFire
12 10 (0.63%) Gunicorn
13 9 (0.57%) Luigi
14 6 (0.38%) Oracle Coherence
15 5 (0.32%) GeoServer
15 5 (0.32%) Oracle Workflow
16 4 (0.25%) Jetty
Applications
1 11 (0.70%) Microsoft Excel
2 9 (0.57%) SPSS
3 8 (0.51%) Chatbot
4 3 (0.19%) Microsoft Office
5 2 (0.13%) Pipeline Pilot
5 2 (0.13%) Weka
6 1 (0.063%) MS Visio
6 1 (0.063%) Shockwave
Business Applications
1 6 (0.38%) SAP Oil and Gas
2 2 (0.13%) Dynamics CRM
3 1 (0.063%) Finacle
3 1 (0.063%) FLEXCUBE
3 1 (0.063%) ION MarketView
3 1 (0.063%) Markit EDM
3 1 (0.063%) Oracle BPM
3 1 (0.063%) PolarLake
Cloud Services
1 391 (24.79%) Amazon AWS
2 142 (9.00%) Microsoft Azure
3 75 (4.76%) Amazon S3
4 69 (4.38%) Google Cloud Platform
5 64 (4.06%) GitHub
6 61 (3.87%) Amazon EMR
7 50 (3.17%) Serverless
8 47 (2.98%) Amazon Kinesis
9 41 (2.60%) AWS Lambda
10 34 (2.16%) PaaS
11 31 (1.97%) Amazon EC2
12 24 (1.52%) Azure Data Factory
13 21 (1.33%) Google Tag Manager
14 18 (1.14%) IaaS
14 18 (1.14%) SaaS
15 17 (1.08%) Google Analytics
16 15 (0.95%) Data Lake Analytics
17 13 (0.82%) Cloud Computing
18 11 (0.70%) Route 53
19 10 (0.63%) IBM Cloud
Communications & Networking
1 90 (5.71%) Kerberos
2 37 (2.35%) HTTP
3 34 (2.16%) SSL
4 18 (1.14%) DNS
5 14 (0.89%) TCP/IP
6 11 (0.70%) VPN
7 9 (0.57%) ICMP
8 8 (0.51%) DSL
8 8 (0.51%) Internet
8 8 (0.51%) Network Security
9 7 (0.44%) Firewall
10 6 (0.38%) OpenLDAP
11 5 (0.32%) FTP
12 3 (0.19%) SSH
13 2 (0.13%) AMQP
13 2 (0.13%) BIG-IP
13 2 (0.13%) SAN
14 1 (0.063%) Multicast
14 1 (0.063%) SMS
14 1 (0.063%) SNA
Database & Business Intelligence
1 1,136 (72.04%) Big Data
2 1,127 (71.46%) Hadoop
3 579 (36.72%) Apache Hive
4 389 (24.67%) HBase
5 335 (21.24%) NoSQL
6 253 (16.04%) Data Warehouse
7 232 (14.71%) Apache Cassandra
8 188 (11.92%) MongoDB
9 181 (11.48%) Impala
10 177 (11.22%) SQL Server
11 164 (10.40%) Sqoop
12 146 (9.26%) PostgreSQL
13 134 (8.50%) Relational Database
14 124 (7.86%) Data Lake
15 111 (7.04%) Amazon Redshift
16 95 (6.02%) MySQL
17 64 (4.06%) RDBMS
18 58 (3.68%) OLAP
19 54 (3.42%) Star Schema
20 53 (3.36%) Cloudera CDH
Development Applications
1 220 (13.95%) Jenkins
2 201 (12.75%) Git (software)
3 52 (3.30%) JIRA
4 39 (2.47%) Maven
5 34 (2.16%) GitLab
6 33 (2.09%) Cucumber
6 33 (2.09%) Subversion
7 29 (1.84%) JUnit
8 27 (1.71%) Team Foundation Server
9 26 (1.65%) Gradle
9 26 (1.65%) TeamCity
10 23 (1.46%) Visual Studio Team System
11 22 (1.40%) Artifactory
11 22 (1.40%) Sonatype Nexus
12 17 (1.08%) Selenium
13 16 (1.01%) SonarQube
14 13 (0.82%) Hudson
14 13 (0.82%) Vagrant
14 13 (0.82%) Visual Studio
15 11 (0.70%) GoCD
General
1 353 (22.38%) Finance
2 178 (11.29%) Banking
3 80 (5.07%) Retail
4 53 (3.36%) Marketing
5 47 (2.98%) Investment Banking
6 44 (2.79%) Telecoms
7 43 (2.73%) Manufacturing
8 34 (2.16%) Games
9 26 (1.65%) Front Office
10 24 (1.52%) Publishing
11 15 (0.95%) Advertising
12 14 (0.89%) Retail Banking
13 12 (0.76%) Automotive
14 11 (0.70%) Electronics
15 7 (0.44%) Private Banking
16 6 (0.38%) Financial Institution
16 6 (0.38%) Pharmaceutical
17 5 (0.32%) Legal
18 4 (0.25%) French Language
18 4 (0.25%) Law
Job Titles
1 529 (33.54%) Developer
2 441 (27.96%) Data Engineer
3 267 (16.93%) Big Data Engineer
4 246 (15.60%) Big Data Developer
5 162 (10.27%) Architect
6 157 (9.96%) Data Scientist
7 138 (8.75%) Java Developer
8 126 (7.99%) Scala Developer
9 125 (7.93%) Hadoop Engineer
10 123 (7.80%) Apache Spark Developer
11 120 (7.61%) Data Architect
12 96 (6.09%) Hadoop Developer
13 89 (5.64%) DevOps Engineer
14 84 (5.33%) Big Data Architect
15 72 (4.57%) AWS Engineer
16 60 (3.80%) Consultant
17 51 (3.23%) Senior Developer
18 49 (3.11%) Python Engineer
18 49 (3.11%) Senior Data Engineer
19 48 (3.04%) Analyst
Libraries, Frameworks & Software Standards
1 690 (43.75%) Kafka
2 165 (10.46%) Spark Streaming
3 148 (9.38%) RESTful
4 100 (6.34%) Spring
5 97 (6.15%) JSON
6 94 (5.96%) RabbitMQ
7 90 (5.71%) REST
8 87 (5.52%) Web Services
9 73 (4.63%) PySpark
10 65 (4.12%) Akka
10 65 (4.12%) TensorFlow
11 61 (3.87%) Node.js
11 61 (3.87%) XML
12 59 (3.74%) Spring Boot
13 52 (3.30%) AngularJS
13 52 (3.30%) J2EE
13 52 (3.30%) Natural Language Processing
14 50 (3.17%) React
15 46 (2.92%) Elastic Stack
16 45 (2.85%) Apache Avro
Miscellaneous
1 163 (10.34%) Algorithms
2 152 (9.64%) Analytical Skills
3 128 (8.12%) Computer Science
4 113 (7.17%) Greenfield Project
5 51 (3.23%) Fintech
6 49 (3.11%) Data Structures
7 38 (2.41%) Self-Motivation
8 30 (1.90%) Derivative
9 28 (1.78%) Linux Command Line
10 27 (1.71%) User Experience
11 24 (1.52%) Virtual Team
12 17 (1.08%) Market Risk
13 12 (0.76%) Data Centre
13 12 (0.76%) Replication
14 11 (0.70%) Grid Computing
14 11 (0.70%) Management Information System
14 11 (0.70%) Public Cloud
15 9 (0.57%) Enterprise Software
16 8 (0.51%) Credit Risk
16 8 (0.51%) HoloLens
Operating Systems
1 345 (21.88%) Linux
2 129 (8.18%) Unix
3 55 (3.49%) Red Hat Enterprise Linux
4 40 (2.54%) CentOS
5 23 (1.46%) Ubuntu
6 22 (1.40%) Windows
7 8 (0.51%) Windows Server
7 8 (0.51%) Windows Server 2008
8 6 (0.38%) CoreOS
8 6 (0.38%) Debian
9 5 (0.32%) Android
9 5 (0.32%) Apple iOS
9 5 (0.32%) DC/OS
10 4 (0.25%) Mac OS X
11 2 (0.13%) SUSE Linux Enterprise Server
12 1 (0.063%) VMS
Processes & Methodologies
1 582 (36.91%) Agile Software Development
2 479 (30.37%) Analytics
3 389 (24.67%) Data Science
4 352 (22.32%) Machine Learning
5 290 (18.39%) DevOps
6 288 (18.26%) ETL
7 217 (13.76%) TDD
8 181 (11.48%) Continuous Integration
9 169 (10.72%) Data Engineering
10 165 (10.46%) Scrum
11 164 (10.40%) MapReduce
12 163 (10.34%) Data Modelling
13 151 (9.58%) Business Intelligence
14 142 (9.00%) Visualisation
15 134 (8.50%) Open Source
16 133 (8.43%) Data Analytics
17 124 (7.86%) Test Automation
18 122 (7.74%) Software Engineering
19 120 (7.61%) Microservices
20 119 (7.55%) Problem-Solving
Programming Languages
1 766 (48.57%) Python
2 675 (42.80%) Java
3 664 (42.11%) Scala
4 507 (32.15%) SQL
5 192 (12.18%) R
6 109 (6.91%) JavaScript
7 89 (5.64%) Shell Script
8 85 (5.39%) Bash Shell
9 70 (4.44%) Java 8
10 50 (3.17%) Ruby
11 38 (2.41%) Perl
12 31 (1.97%) C++
13 29 (1.84%) C
14 28 (1.78%) C#
15 23 (1.46%) Groovy
15 23 (1.46%) PL/SQL
16 22 (1.40%) Go
17 19 (1.20%) MATLAB
18 16 (1.01%) T-SQL
19 12 (0.76%) PowerShell
Qualifications
1 132 (8.37%) Degree
2 96 (6.09%) Security Cleared
3 79 (5.01%) SC Cleared
4 36 (2.28%) Computer Science Degree
5 34 (2.16%) PhD
6 23 (1.46%) Master's Degree
7 17 (1.08%) Numerate Degree
8 10 (0.63%) DV Cleared
9 6 (0.38%) ISTQB
9 6 (0.38%) Microsoft Certification
10 5 (0.32%) CTC Cleared
11 3 (0.19%) AWS Certification
12 2 (0.13%) TOGAF Certification
13 1 (0.063%) ACCA
13 1 (0.063%) CIPD
13 1 (0.063%) ISTQB Foundation Certification
13 1 (0.063%) MBA
13 1 (0.063%) MCSD
13 1 (0.063%) PRINCE2 Certification
Quality Assurance & Compliance
1 98 (6.21%) Data Quality
2 68 (4.31%) QA
3 23 (1.46%) GCP
4 20 (1.27%) GDPR
5 6 (0.38%) PMO
6 3 (0.19%) SLA
7 1 (0.063%) Basel II
7 1 (0.063%) Disclosure Scotland
7 1 (0.063%) FATCA
7 1 (0.063%) ISO/IEC 27001
7 1 (0.063%) NIST 800
7 1 (0.063%) PSD2
System Software
1 272 (17.25%) HDFS
2 205 (13.00%) Apache Flume
3 202 (12.81%) Docker
4 122 (7.74%) Apache ZooKeeper
5 22 (1.40%) Active Directory
6 10 (0.63%) Virtual Servers
7 5 (0.32%) Squid
7 5 (0.32%) Virtual Machines
8 2 (0.13%) Varnish
9 1 (0.063%) etcd
9 1 (0.063%) VirtualBox
9 1 (0.063%) VMware Infrastructure
Systems Management
1 227 (14.39%) Oozie
2 165 (10.46%) Hadoop YARN
3 108 (6.85%) Ansible
4 79 (5.01%) Puppet
5 72 (4.57%) Kubernetes
6 60 (3.80%) Terraform
7 52 (3.30%) Ambari
8 39 (2.47%) Mesos
9 34 (2.16%) Opscode Chef
10 27 (1.71%) Kibana
11 20 (1.27%) Apache Slider
12 16 (1.01%) Consul
12 16 (1.01%) Salt
13 15 (0.95%) ITRS Geneos
14 14 (0.89%) logstash
15 12 (0.76%) Packer
16 11 (0.70%) Prometheus
17 10 (0.63%) Sysdig
18 7 (0.44%) Marathon
18 7 (0.44%) Nagios
Vendors
1 287 (18.20%) Cloudera
2 174 (11.03%) Hortonworks
3 167 (10.59%) Oracle
4 100 (6.34%) Talend
5 99 (6.28%) Google
6 86 (5.45%) Microsoft
7 83 (5.26%) Tableau
8 80 (5.07%) MapR
9 74 (4.69%) Informatica
10 50 (3.17%) SAP
10 50 (3.17%) Splunk
11 45 (2.85%) Salesforce.com
12 37 (2.35%) Alteryx
13 36 (2.28%) IBM
14 23 (1.46%) Red Hat
15 21 (1.33%) Tealium
16 18 (1.14%) Teradata
17 15 (0.95%) AppDynamics
17 15 (0.95%) Atlassian
18 13 (0.82%) SaltStack