Period
to 24 September 2018

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 24 September 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
24 Sep 2018
Same period 2017 Same period 2016
Rank 188 194 304
Rank change year-on-year +6 +110 +271
Contract jobs citing Apache Spark 1,499 1,352 895
As % of all contract IT jobs advertised in the UK 1.47% 1.39% 0.84%
As % of the Application Platforms category 8.76% 8.27% 4.51%
Number of daily rates quoted 1,079 953 625
UK median daily rate £550 £550 £550
10th Percentile £425 £400 £400
90th Percentile £700 £688 £688
UK excluding London median daily rate £500 £463 £425
% change year-on-year +8.11% +8.82% -5.56%
Number of hourly rates quoted 8 5 2
UK median hourly rate £33.75 £19.34 £20.50
Median hourly rate % change year-on-year +74.51% -5.66% -38.90%
10th Percentile £16.48 £16.64 £19.25
90th Percentile £73.75 £42.50 £21.75
UK excluding London median hourly rate £22.00 £19.34 £20.50
% change year-on-year +13.75% -5.66% -38.90%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 17,109 16,351 19,830
As % of all contract IT jobs advertised in the UK 16.78% 16.79% 18.59%
Number of daily rates quoted 11,211 10,610 13,079
UK median daily rate £450 £438 £420
Median daily rate % change year-on-year +2.86% +4.17% +5.00%
10th Percentile £270 £263 £253
90th Percentile £625 £600 £588
UK excluding London median daily rate £425 £400 £390
% change year-on-year +6.25% +2.56% +4.00%
Number of hourly rates quoted 423 414 498
UK median hourly rate £21.00 £20.00 £20.00
Median hourly rate % change year-on-year +5.00% - -
10th Percentile £12.00 £11.00 £12.00
90th Percentile £51.00 £48.75 £45.00
UK excluding London median hourly rate £20.50 £20.00 £20.73
% change year-on-year +2.50% -3.52% +3.65%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 24 September 2018.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 24 September 2018.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 24 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -8 1,433 £550 - 241
London -13 960 £550 - 158
UK excluding London +54 525 £500 +8.11% 105
South East +32 220 £500 +5.26% 37
North of England +6 103 £525 +14.13% 11
South West +36 63 £525 +24.26% 12
East of England -17 54 £500 +2.56% 8
Yorkshire -12 51 £475 +5.56%
North West +4 49 £550 +4.76% 11
Scotland -9 35 £450 +2.86% 13
Midlands -28 25 £466 +3.61% 15
East Midlands +22 20 £473 +11.18% 11
Northern Ireland +50 17 £432 - 4
Wales -22 8 £500 +53.85% 5
West Midlands -47 5 £375 -16.67% 4
North East -11 3 £625 +47.06%

For the 6 months to 24 September 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 1,048 (69.91%) Big Data
2 1,047 (69.85%) Hadoop
3 736 (49.10%) Python
4 728 (48.57%) Kafka
5 681 (45.43%) Java
6 542 (36.16%) Agile Software Development
7 518 (34.56%) Apache Hive
8 515 (34.36%) SQL
9 508 (33.89%) Scala
10 464 (30.95%) Analytics
11 452 (30.15%) Finance
12 396 (26.42%) Amazon AWS
13 385 (25.68%) Data Science
14 329 (21.95%) NoSQL
15 327 (21.81%) HBase
16 314 (20.95%) Apache
17 295 (19.68%) Linux
18 289 (19.28%) Machine Learning
19 244 (16.28%) DevOps
20 241 (16.08%) TDD
21 231 (15.41%) ETL
22 229 (15.28%) Apache Cassandra
23 227 (15.14%) Data Warehouse
24 225 (15.01%) Git (software)
24 225 (15.01%) HDFS
25 205 (13.68%) Jenkins
26 204 (13.61%) Microsoft Azure
27 196 (13.08%) Docker
28 192 (12.81%) MongoDB
29 191 (12.74%) Elasticsearch

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 314 (20.95%) Apache
2 191 (12.74%) Elasticsearch
3 97 (6.47%) Apache Pig
4 79 (5.27%) Apache Solr
5 50 (3.34%) Apache Airflow
6 45 (3.00%) SAS
7 41 (2.74%) Confluence
8 23 (1.53%) nginx
9 17 (1.13%) Luigi
10 15 (1.00%) SharePoint
11 14 (0.93%) Cloud Foundry
12 13 (0.87%) GemFire
13 11 (0.73%) Gunicorn
14 9 (0.60%) Tomcat
15 8 (0.53%) Oracle Workflow
16 7 (0.47%) Jupyter
17 5 (0.33%) Oracle Coherence
17 5 (0.33%) Oracle SOA Suite
18 3 (0.20%) BizTalk Server
19 2 (0.13%) OpenStack
Applications
1 9 (0.60%) Microsoft Excel
2 8 (0.53%) Chatbot
3 3 (0.20%) MS Visio
4 2 (0.13%) Microsoft PowerPoint
4 2 (0.13%) Microsoft Project
5 1 (0.067%) GNU Octave
5 1 (0.067%) InVision
5 1 (0.067%) Microsoft Office
5 1 (0.067%) Photoshop
5 1 (0.067%) Shockwave
5 1 (0.067%) Spreadsheet
5 1 (0.067%) SPSS
5 1 (0.067%) Stata
Business Applications
1 17 (1.13%) SAP Oil and Gas
2 3 (0.20%) Dynamics AX
2 3 (0.20%) Dynamics CRM
2 3 (0.20%) Oracle Financials
2 3 (0.20%) SAP MM
2 3 (0.20%) SAP R/3
3 2 (0.13%) Markit EDM
3 2 (0.13%) PolarLake
4 1 (0.067%) Finacle
4 1 (0.067%) FLEXCUBE
4 1 (0.067%) ION MarketView
4 1 (0.067%) Remedy ITSM
4 1 (0.067%) Salesforce.com CRM
Cloud Services
1 396 (26.42%) Amazon AWS
2 204 (13.61%) Microsoft Azure
3 111 (7.40%) Google Cloud Platform
4 88 (5.87%) Amazon EMR
5 62 (4.14%) Amazon S3
6 60 (4.00%) Amazon Kinesis
7 48 (3.20%) GitHub
8 38 (2.54%) Amazon EC2
9 34 (2.27%) Serverless
10 28 (1.87%) Cloud Computing
11 26 (1.73%) AWS Lambda
11 26 (1.73%) PaaS
12 25 (1.67%) Azure Data Factory
13 23 (1.53%) Heroku
14 16 (1.07%) Azure Machine Learning
15 10 (0.67%) Route 53
16 9 (0.60%) Data Lake Analytics
16 9 (0.60%) IBM Cloud
16 9 (0.60%) OpenShift
17 8 (0.53%) SaaS
Communications & Networking
1 66 (4.40%) Kerberos
2 55 (3.67%) HTTP
3 44 (2.94%) SSL
4 38 (2.54%) TCP/IP
5 30 (2.00%) DNS
6 25 (1.67%) FTP
7 21 (1.40%) Internet
8 20 (1.33%) Firewall
8 20 (1.33%) VPN
9 14 (0.93%) SSH
10 7 (0.47%) HTTPS
11 5 (0.33%) DAB
11 5 (0.33%) SNA
12 2 (0.13%) RTMP
13 1 (0.067%) Broadband
13 1 (0.067%) DHCP
13 1 (0.067%) DSL
13 1 (0.067%) ICMP
13 1 (0.067%) IPsec
13 1 (0.067%) SMS
Database & Business Intelligence
1 1,048 (69.91%) Big Data
2 1,047 (69.85%) Hadoop
3 518 (34.56%) Apache Hive
4 329 (21.95%) NoSQL
5 327 (21.81%) HBase
6 229 (15.28%) Apache Cassandra
7 227 (15.14%) Data Warehouse
8 192 (12.81%) MongoDB
9 188 (12.54%) Relational Database
10 152 (10.14%) SQL Server
11 140 (9.34%) PostgreSQL
12 125 (8.34%) Impala
13 122 (8.14%) Amazon Redshift
14 119 (7.94%) Data Lake
15 89 (5.94%) Sqoop
16 87 (5.80%) RDBMS
17 75 (5.00%) MySQL
18 71 (4.74%) Enterprise Data Warehouse
18 71 (4.74%) OLAP
19 70 (4.67%) Star Schema
Development Applications
1 225 (15.01%) Git (software)
2 205 (13.68%) Jenkins
3 78 (5.20%) JIRA
4 45 (3.00%) Visual Studio Team System
5 43 (2.87%) GitLab
6 40 (2.67%) Cucumber
7 37 (2.47%) Maven
8 31 (2.07%) Subversion
9 24 (1.60%) Atlassian Bamboo
10 22 (1.47%) IntelliJ
11 21 (1.40%) Sonatype Nexus
12 18 (1.20%) Gradle
12 18 (1.20%) JUnit
12 18 (1.20%) Selenium
13 17 (1.13%) Bitbucket
14 16 (1.07%) gulp
15 13 (0.87%) Visual Studio
16 11 (0.73%) Travis CI
17 10 (0.67%) Eclipse
17 10 (0.67%) Mercurial
General
1 452 (30.15%) Finance
2 132 (8.81%) Banking
3 47 (3.14%) Retail
4 40 (2.67%) Front Office
5 39 (2.60%) Investment Banking
6 34 (2.27%) Legal
7 33 (2.20%) Games
8 30 (2.00%) Marketing
8 30 (2.00%) Telecoms
9 17 (1.13%) Publishing
10 14 (0.93%) Manufacturing
10 14 (0.93%) Retail Banking
11 13 (0.87%) Advertising
12 6 (0.40%) Prime Brokerage
13 5 (0.33%) Financial Institution
14 4 (0.27%) Law
15 3 (0.20%) International Banking
16 2 (0.13%) Back Office
16 2 (0.13%) Electronics
16 2 (0.13%) Wholesale Banking
Job Titles
1 569 (37.96%) Developer
2 458 (30.55%) Data Engineer
3 265 (17.68%) Big Data Engineer
4 205 (13.68%) Big Data Developer
5 173 (11.54%) Java Developer
6 138 (9.21%) Hadoop Engineer
7 135 (9.01%) Architect
8 132 (8.81%) Apache Spark Developer
9 91 (6.07%) Data Architect
10 88 (5.87%) Python Developer
11 81 (5.40%) Senior Data Engineer
12 79 (5.27%) Data Scientist
13 76 (5.07%) Hadoop Developer
14 72 (4.80%) AWS Engineer
14 72 (4.80%) Scala Developer
15 67 (4.47%) Big Data Architect
16 66 (4.40%) Senior Developer
17 59 (3.94%) Python Engineer
18 55 (3.67%) Software Engineer
19 54 (3.60%) DevOps Engineer
Libraries, Frameworks & Software Standards
1 728 (48.57%) Kafka
2 138 (9.21%) Spark Streaming
3 126 (8.41%) Spring
4 115 (7.67%) REST
5 114 (7.61%) RabbitMQ
6 96 (6.40%) JSON
7 94 (6.27%) RESTful
8 76 (5.07%) AngularJS
9 71 (4.74%) Spring Boot
10 66 (4.40%) J2EE
11 62 (4.14%) HTML
12 56 (3.74%) Natural Language Processing
12 56 (3.74%) Pandas
13 55 (3.67%) PySpark
14 54 (3.60%) React
15 47 (3.14%) HTML5
15 47 (3.14%) Middleware
15 47 (3.14%) TensorFlow
16 46 (3.07%) Node.js
17 44 (2.94%) NumPy
Miscellaneous
1 155 (10.34%) Algorithms
2 113 (7.54%) Computer Science
3 110 (7.34%) Analytical Skills
4 99 (6.60%) Greenfield Project
5 54 (3.60%) Fintech
6 46 (3.07%) Data Structures
7 31 (2.07%) Foreign Exchange (FX)
8 26 (1.73%) Self-Motivation
9 25 (1.67%) Enterprise Software
10 19 (1.27%) HoloLens
11 16 (1.07%) Credit Risk
11 16 (1.07%) Linux Command Line
12 15 (1.00%) Derivative
12 15 (1.00%) Fixed Income
12 15 (1.00%) Virtual Team
13 13 (0.87%) User Experience
14 9 (0.60%) Grid Computing
14 9 (0.60%) Market Risk
15 7 (0.47%) Data Centre
15 7 (0.47%) Internet of Things
Operating Systems
1 295 (19.68%) Linux
2 149 (9.94%) Unix
3 17 (1.13%) Windows
4 16 (1.07%) Red Hat Enterprise Linux
5 14 (0.93%) Ubuntu
6 5 (0.33%) DC/OS
7 4 (0.27%) CentOS
8 2 (0.13%) Apple iOS
8 2 (0.13%) SUSE Linux Enterprise Server
9 1 (0.067%) Android
9 1 (0.067%) Mac OS X
9 1 (0.067%) Oracle Linux
9 1 (0.067%) Windows Server
9 1 (0.067%) Windows Server 2008
Processes & Methodologies
1 542 (36.16%) Agile Software Development
2 464 (30.95%) Analytics
3 385 (25.68%) Data Science
4 289 (19.28%) Machine Learning
5 244 (16.28%) DevOps
6 241 (16.08%) TDD
7 231 (15.41%) ETL
8 191 (12.74%) Data Modelling
9 186 (12.41%) Scrum
10 185 (12.34%) Business Intelligence
11 157 (10.47%) Continuous Integration
12 154 (10.27%) Open Source
13 152 (10.14%) BDD
14 139 (9.27%) Software Engineering
15 134 (8.94%) Visualisation
16 130 (8.67%) Data Engineering
17 121 (8.07%) Data Analytics
18 118 (7.87%) Microservices
19 116 (7.74%) MapReduce
20 95 (6.34%) OO
Programming Languages
1 736 (49.10%) Python
2 681 (45.43%) Java
3 515 (34.36%) SQL
4 508 (33.89%) Scala
5 131 (8.74%) R
6 118 (7.87%) Java 8
7 110 (7.34%) JavaScript
8 71 (4.74%) Bash Shell
9 66 (4.40%) C++
10 59 (3.94%) Shell Script
11 46 (3.07%) C#
12 38 (2.54%) Ruby
13 35 (2.33%) MATLAB
14 34 (2.27%) Perl
15 30 (2.00%) TypeScript
16 26 (1.73%) ES6
17 23 (1.53%) C
18 22 (1.47%) PL/SQL
19 15 (1.00%) T-SQL
20 14 (0.93%) Go
Qualifications
1 110 (7.34%) Degree
2 63 (4.20%) Security Cleared
3 52 (3.47%) Computer Science Degree
4 47 (3.14%) SC Cleared
5 29 (1.93%) PhD
6 18 (1.20%) Master's Degree
7 16 (1.07%) ISTQB
8 8 (0.53%) Microsoft Certification
9 7 (0.47%) DV Cleared
9 7 (0.47%) Numerate Degree
10 4 (0.27%) AWS Certification
11 3 (0.20%) BPSS Clearance
11 3 (0.20%) CCA
11 3 (0.20%) ISEB
12 2 (0.13%) Sun Certification
13 1 (0.067%) CIPD
13 1 (0.067%) CISSP
13 1 (0.067%) MBA
13 1 (0.067%) MCSA
13 1 (0.067%) MCSE
Quality Assurance & Compliance
1 107 (7.14%) Data Quality
2 33 (2.20%) QA
3 23 (1.53%) GDPR
4 10 (0.67%) ISO/IEC 27001
5 4 (0.27%) EMIR
5 4 (0.27%) MiFID
6 3 (0.20%) SLA
7 2 (0.13%) NIST 800
7 2 (0.13%) PMO
8 1 (0.067%) HL7
8 1 (0.067%) PSD2
System Software
1 225 (15.01%) HDFS
2 196 (13.08%) Docker
3 89 (5.94%) Apache Flume
4 82 (5.47%) Apache ZooKeeper
5 16 (1.07%) Active Directory
6 3 (0.20%) Virtual Machines
6 3 (0.20%) Virtual Servers
7 2 (0.13%) Ceph
Systems Management
1 172 (11.47%) Oozie
2 140 (9.34%) Hadoop YARN
3 116 (7.74%) Ansible
3 116 (7.74%) Kubernetes
4 74 (4.94%) Terraform
5 60 (4.00%) Puppet
6 43 (2.87%) Opscode Chef
7 32 (2.13%) Kibana
8 28 (1.87%) Mesos
9 23 (1.53%) Ambari
10 20 (1.33%) logstash
11 19 (1.27%) Docker Swarm
12 15 (1.00%) Marathon
13 9 (0.60%) Salt
14 7 (0.47%) Packer
14 7 (0.47%) Single Sign-On
15 6 (0.40%) Consul
16 5 (0.33%) Apache Slider
16 5 (0.33%) Prometheus
17 3 (0.20%) ITRS Geneos
Vendors
1 147 (9.81%) Oracle
2 129 (8.61%) Cloudera
3 89 (5.94%) Hortonworks
4 88 (5.87%) Google
5 85 (5.67%) Tableau
6 78 (5.20%) Microsoft
7 58 (3.87%) Splunk
8 55 (3.67%) Informatica
9 52 (3.47%) Talend
10 36 (2.40%) Alteryx
10 36 (2.40%) MapR
11 35 (2.33%) SAP
12 30 (2.00%) IBM
13 28 (1.87%) Databricks
14 15 (1.00%) SaltStack
15 14 (0.93%) Ab Initio
15 14 (0.93%) Teradata
16 13 (0.87%) Atlassian
17 12 (0.80%) Red Hat
17 12 (0.80%) Sybase