Period
to 24 September 2017

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 24 September 2017 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
24 Sep 2017
Same period 2016 Same period 2015
Rank 192 299 567
Rank change year-on-year +107 +268 +238
Contract jobs citing Apache Spark 1351 895 363
As % of all contract IT jobs advertised in the UK 1.39% 0.84% 0.32%
As % of the Application Platforms category 8.29% 4.52% 1.77%
Number of daily rates quoted 952 625 271
UK median daily rate £550 £550 £600
Median daily rate % change year-on-year - -8.33% +5.49%
10th Percentile £400 £400 £425
90th Percentile £688 £688 £725
UK excluding London median daily rate £463 £425 £450
% change year-on-year +8.82% -5.56% -21.74%
Number of hourly rates quoted 5 2 3
UK median hourly rate £19.34 £20.50 £33.55
Median hourly rate % change year-on-year -5.66% -38.90% -
UK excluding London median hourly rate £19.34 £20.50 £33.55
% change year-on-year -5.66% -38.90% -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 16293 19800 20539
As % of all contract IT jobs advertised in the UK 16.74% 18.56% 17.99%
Number of daily rates quoted 10568 13056 12926
UK median daily rate £438 £420 £400
Median daily rate % change year-on-year +4.17% +5.00% +6.67%
10th Percentile £263 £253 £225
90th Percentile £600 £588 £563
UK excluding London median daily rate £400 £390 £375
% change year-on-year +2.56% +4.00% +7.14%
Number of hourly rates quoted 414 498 674
UK median hourly rate £20.00 £20.00 £20.00
10th Percentile £11.00 £12.00 £11.25
90th Percentile £48.75 £45.00 £42.02
UK excluding London median hourly rate £20.00 £20.73 £20.00
% change year-on-year -3.52% +3.65% +14.29%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 24 September 2017.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 24 September 2017.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 24 September 2017. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +113 1311 £550 - 96
London +59 947 £550 - 66
UK excluding London +142 390 £463 +8.82% 32
South East +53 164 £475 -8.48% 9
North of England +110 72 £460 +6.67% 7
East of England +42 68 £488 +12.72% 4
Yorkshire +59 43 £450 +5.88% 2
Midlands +56 36 £450 +5.88% 4
West Midlands +59 30 £450 - 3
North West +40 29 £525 +31.25% 5
Scotland +46 25 £438 -2.89% 1
South West -20 18 £423 +30.00% 6
East Midlands +2 6 £425 - 1
Wales -10 4 £325 -44.68% 1
Northern Ireland - 3 - -
North East +75 2 £425 -15.00%

For the 6 months to 24 September 2017, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the number co-occurrences and its proportion to all contract ads with a requirement for Apache Spark.

1 1054 (78.02%) Hadoop
2 1025 (75.87%) Big Data
3 769 (56.92%) Java
4 557 (41.23%) Python
5 546 (40.41%) Scala
6 535 (39.60%) Agile Software Development
7 530 (39.23%) Kafka
8 507 (37.53%) Apache Hive
9 406 (30.05%) SQL
10 358 (26.50%) HBase
11 349 (25.83%) Analytics
12 316 (23.39%) Finance
13 308 (22.80%) HDFS
14 304 (22.50%) TDD
15 289 (21.39%) Linux
16 264 (19.54%) MapReduce
17 261 (19.32%) Apache
18 260 (19.25%) NoSQL
19 243 (17.99%) Amazon AWS
20 240 (17.76%) Scrum
20 240 (17.76%) Machine Learning
21 235 (17.39%) Apache Cassandra
22 216 (15.99%) ETL
23 215 (15.91%) Cloudera
24 214 (15.84%) DevOps
25 208 (15.40%) Banking
26 198 (14.66%) Impala
27 191 (14.14%) Apache Flume
28 186 (13.77%) Spring
29 185 (13.69%) Unix

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 261 (19.32%) Apache
2 119 (8.81%) Elasticsearch
3 117 (8.66%) Apache Solr
4 111 (8.22%) Apache Pig
5 51 (3.77%) Confluence
6 44 (3.26%) SAS
7 37 (2.74%) Tomcat
8 21 (1.55%) JBoss
9 16 (1.18%) Cloud Foundry
10 13 (0.96%) OpenStack
11 12 (0.89%) Apache Airflow
11 12 (0.89%) Oracle SOA Suite
12 11 (0.81%) GemFire
12 11 (0.81%) GlassFish
12 11 (0.81%) WildFly
13 9 (0.67%) nginx
14 6 (0.44%) Jupyter
14 6 (0.44%) Luigi
15 4 (0.30%) Oracle Coherence
15 4 (0.30%) WebLogic
Applications
1 18 (1.33%) SPSS
2 12 (0.89%) Chatbot
3 7 (0.52%) Stata
4 5 (0.37%) Microsoft Excel
4 5 (0.37%) Microsoft Office
4 5 (0.37%) Pipeline Pilot
5 1 (0.074%) GNU Octave
5 1 (0.074%) Microsoft PowerPoint
5 1 (0.074%) Spreadsheet
5 1 (0.074%) Weka
Business Applications
1 1 (0.074%) SAP CRM
Cloud Services
1 243 (17.99%) Amazon AWS
2 97 (7.18%) GitHub
3 67 (4.96%) Microsoft Azure
4 51 (3.77%) Google Cloud Platform
5 34 (2.52%) Bluemix
5 34 (2.52%) IBM Cloud
6 26 (1.92%) Amazon EMR
7 22 (1.63%) Amazon EC2
7 22 (1.63%) Amazon S3
8 20 (1.48%) Virtual Private Cloud
9 15 (1.11%) PaaS
10 12 (0.89%) Cloud Computing
10 12 (0.89%) SaaS
11 9 (0.67%) Amazon Kinesis
11 9 (0.67%) Google Analytics
12 5 (0.37%) Amazon Glacier
13 4 (0.30%) AWS Lambda
13 4 (0.30%) Azure Machine Learning
13 4 (0.30%) IaaS
14 3 (0.22%) Azure Data Factory
Communications & Networking
1 117 (8.66%) Kerberos
2 21 (1.55%) Firewall
3 17 (1.26%) HTTP
4 16 (1.18%) DNS
4 16 (1.18%) VPN
5 15 (1.11%) LAN
6 14 (1.04%) WAN
7 8 (0.59%) LDAP
8 5 (0.37%) Network Security
9 4 (0.30%) Skype
10 3 (0.22%) Intranet
10 3 (0.22%) OpenLDAP
10 3 (0.22%) SNA
11 2 (0.15%) ICMP
11 2 (0.15%) Internet
11 2 (0.15%) TCP/IP
12 1 (0.074%) Broadband
Database & Business Intelligence
1 1054 (78.02%) Hadoop
2 1025 (75.87%) Big Data
3 507 (37.53%) Apache Hive
4 358 (26.50%) HBase
5 260 (19.25%) NoSQL
6 235 (17.39%) Apache Cassandra
7 198 (14.66%) Impala
8 170 (12.58%) MongoDB
8 170 (12.58%) Sqoop
9 133 (9.84%) PostgreSQL
10 114 (8.44%) Data Warehouse
11 92 (6.81%) Relational Database
12 68 (5.03%) SQL Server
13 55 (4.07%) MySQL
14 39 (2.89%) Amazon Redshift
15 36 (2.66%) Data Lake
16 35 (2.59%) Data Mining
17 32 (2.37%) Flink
17 32 (2.37%) RDBMS
18 28 (2.07%) Redis
Development Applications
1 171 (12.66%) Git (software)
2 165 (12.21%) Jenkins
3 123 (9.10%) JIRA
4 78 (5.77%) Cucumber
5 44 (3.26%) Maven
6 41 (3.03%) JUnit
7 30 (2.22%) Gradle
8 28 (2.07%) Vagrant
9 27 (2.00%) Hudson
9 27 (2.00%) Mockito
10 24 (1.78%) Talend Open Studio
11 21 (1.55%) IntelliJ
12 20 (1.48%) Subversion
13 19 (1.41%) Team Foundation Server
14 16 (1.18%) NUnit
15 13 (0.96%) Sonatype Nexus
16 12 (0.89%) Eclipse
16 12 (0.89%) TeamCity
17 11 (0.81%) GitLab
17 11 (0.81%) JBehave
General
1 316 (23.39%) Finance
2 208 (15.40%) Banking
3 103 (7.62%) Investment Banking
4 73 (5.40%) Telecoms
5 62 (4.59%) Marketing
6 55 (4.07%) Front Office
7 48 (3.55%) Retail
8 38 (2.81%) Electronics
9 37 (2.74%) Manufacturing
10 18 (1.33%) Games
11 11 (0.81%) Publishing
12 7 (0.52%) Pharmaceutical
13 5 (0.37%) Financial Institution
14 4 (0.30%) Advertising
14 4 (0.30%) Law
14 4 (0.30%) Legal
15 3 (0.22%) Spanish Language
16 2 (0.15%) Automotive
16 2 (0.15%) Back Office
16 2 (0.15%) Private Banking
Job Titles
1 542 (40.12%) Developer
2 324 (23.98%) Data Engineer
3 254 (18.80%) Big Data Engineer
4 244 (18.06%) Big Data Developer
5 198 (14.66%) Java Developer
6 160 (11.84%) Scala Developer
7 119 (8.81%) Hadoop Engineer
8 116 (8.59%) Architect
9 109 (8.07%) Hadoop Developer
10 97 (7.18%) Apache Spark Developer
11 93 (6.88%) Data Scientist
12 77 (5.70%) Java Engineer
13 65 (4.81%) Data Architect
14 53 (3.92%) Big Data Architect
14 53 (3.92%) DevOps Engineer
15 52 (3.85%) Senior Developer
16 43 (3.18%) Java Scala Developer
17 40 (2.96%) Analyst
18 39 (2.89%) Cloud Engineer
19 38 (2.81%) Java Hadoop Developer
Libraries, Frameworks & Software Standards
1 530 (39.23%) Kafka
2 186 (13.77%) Spring
3 145 (10.73%) REST
4 121 (8.96%) Spring Boot
5 108 (7.99%) Spark Streaming
6 96 (7.11%) Akka
7 92 (6.81%) RESTful
8 87 (6.44%) Web Services
9 82 (6.07%) JSON
10 76 (5.63%) Drools
11 52 (3.85%) Apache Camel
11 52 (3.85%) PySpark
12 48 (3.55%) Play Framework
13 44 (3.26%) AngularJS
14 43 (3.18%) Natural Language Processing
14 43 (3.18%) XML
15 41 (3.03%) Hibernate
15 41 (3.03%) HTML
16 37 (2.74%) Node.js
17 34 (2.52%) RabbitMQ
Miscellaneous
1 148 (10.95%) Algorithms
2 144 (10.66%) Analytical Skills
3 54 (4.00%) Computer Science
4 52 (3.85%) Greenfield Project
5 26 (1.92%) Foreign Exchange (FX)
6 22 (1.63%) Market Risk
7 20 (1.48%) Data Structures
7 20 (1.48%) Replication
8 19 (1.41%) User Experience
9 18 (1.33%) Fintech
10 17 (1.26%) Derivative
11 15 (1.11%) Linux Command Line
12 12 (0.89%) Equities
12 12 (0.89%) Mobile App
13 11 (0.81%) Virtual Team
14 10 (0.74%) Credit Risk
14 10 (0.74%) Management Information System
15 7 (0.52%) Fixed Income
15 7 (0.52%) Linked Data
15 7 (0.52%) Semantic Web
Operating Systems
1 289 (21.39%) Linux
2 185 (13.69%) Unix
3 49 (3.63%) Red Hat Enterprise Linux
4 20 (1.48%) CentOS
5 17 (1.26%) Apple iOS
6 16 (1.18%) Android
6 16 (1.18%) Ubuntu
7 15 (1.11%) Windows
8 3 (0.22%) CoreOS
9 1 (0.074%) Debian
9 1 (0.074%) Mac OS X
9 1 (0.074%) Windows 10
9 1 (0.074%) Windows Server
9 1 (0.074%) Windows Server 2008
Processes & Methodologies
1 535 (39.60%) Agile Software Development
2 349 (25.83%) Analytics
3 304 (22.50%) TDD
4 264 (19.54%) MapReduce
5 240 (17.76%) Machine Learning
5 240 (17.76%) Scrum
6 216 (15.99%) ETL
7 214 (15.84%) DevOps
8 181 (13.40%) BDD
9 170 (12.58%) Open Source
10 162 (11.99%) Microservices
11 149 (11.03%) Software Engineering
12 147 (10.88%) Continuous Integration
13 136 (10.07%) Data Architecture
14 135 (9.99%) Data Modelling
15 122 (9.03%) Business Intelligence
16 116 (8.59%) Data Analytics
17 81 (6.00%) Visualisation
18 78 (5.77%) Data Engineering
19 74 (5.48%) Multithreading
Programming Languages
1 769 (56.92%) Java
2 557 (41.23%) Python
3 546 (40.41%) Scala
4 406 (30.05%) SQL
5 165 (12.21%) R
6 113 (8.36%) Shell Script
7 84 (6.22%) JavaScript
8 54 (4.00%) C#
9 42 (3.11%) Perl
10 40 (2.96%) Bash Shell
11 29 (2.15%) Ruby
12 25 (1.85%) MATLAB
13 18 (1.33%) C++
13 18 (1.33%) SPARQL
14 16 (1.18%) PL/SQL
15 14 (1.04%) T-SQL
16 13 (0.96%) Clojure
16 13 (0.96%) Haskell
17 12 (0.89%) Go
17 12 (0.89%) Groovy
Qualifications
1 116 (8.59%) Security Cleared
2 98 (7.25%) SC Cleared
3 71 (5.26%) Degree
4 23 (1.70%) Computer Science Degree
5 21 (1.55%) PhD
6 15 (1.11%) CISSP
7 13 (0.96%) SANS
8 12 (0.89%) CEH
9 9 (0.67%) DV Cleared
10 6 (0.44%) ISTQB
10 6 (0.44%) PMP
11 5 (0.37%) DBS Check
11 5 (0.37%) Numerate Degree
12 4 (0.30%) BPSS Clearance
12 4 (0.30%) ISTQB Foundation Certification
12 4 (0.30%) Master's Degree
13 3 (0.22%) CTC Cleared
14 2 (0.15%) CISA
14 2 (0.15%) CISM
14 2 (0.15%) PMI Certification
Quality Assurance & Compliance
1 79 (5.85%) QA
2 36 (2.66%) Data Quality
3 13 (0.96%) MiFID
4 12 (0.89%) GDPR
5 6 (0.44%) GCP
6 5 (0.37%) Actionable Recommendations
6 5 (0.37%) Dodd-Frank
7 3 (0.22%) IFRS
7 3 (0.22%) IFRS 9
8 2 (0.15%) FedRAMP
8 2 (0.15%) PCI DSS
9 1 (0.074%) Basel III
9 1 (0.074%) EMIR
System Software
1 308 (22.80%) HDFS
2 191 (14.14%) Apache Flume
3 183 (13.55%) Docker
4 97 (7.18%) Apache ZooKeeper
5 11 (0.81%) Active Directory
6 7 (0.52%) Virtual Machines
7 4 (0.30%) Virtual Servers
8 3 (0.22%) Squid
9 1 (0.074%) KVM
Systems Management
1 156 (11.55%) Hadoop YARN
2 122 (9.03%) Ansible
3 117 (8.66%) Oozie
4 93 (6.88%) Kubernetes
5 92 (6.81%) Terraform
6 68 (5.03%) Ambari
7 52 (3.85%) Puppet
8 31 (2.29%) Opscode Chef
9 24 (1.78%) Mesos
10 19 (1.41%) Prometheus
11 17 (1.26%) Kibana
12 13 (0.96%) Control-M
13 12 (0.89%) logstash
14 6 (0.44%) ITRS Geneos
14 6 (0.44%) Packer
15 5 (0.37%) Marathon
15 5 (0.37%) yum
16 3 (0.22%) Consul
16 3 (0.22%) rkt
16 3 (0.22%) Sysdig
Vendors
1 215 (15.91%) Cloudera
2 102 (7.55%) Hortonworks
3 101 (7.48%) Oracle
4 93 (6.88%) Tableau
5 91 (6.74%) Google
6 67 (4.96%) IBM
7 53 (3.92%) Red Hat
8 49 (3.63%) SAP
9 37 (2.74%) Informatica
10 35 (2.59%) Microsoft
11 29 (2.15%) Splunk
12 24 (1.78%) Talend
13 19 (1.41%) Teradata
14 18 (1.33%) Atlassian
14 18 (1.33%) Salesforce.com
15 16 (1.18%) MapR
16 12 (0.89%) Ab Initio
17 11 (0.81%) Sybase
18 10 (0.74%) Netezza
19 6 (0.44%) RapidMiner