Period
to 16 January 2019

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 16 January 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
16 Jan 2019
Same period 2018 Same period 2017
Rank 199 176 216
Rank change year-on-year -23 +40 +240
Contract jobs citing Apache Spark 1,428 1,498 1,208
As % of all contract IT jobs advertised in the UK 1.44% 1.49% 1.25%
As % of the Application Platforms category 8.71% 9.05% 6.88%
Number of daily rates quoted 1,002 1,110 858
UK median daily rate £550 £550 £550
10th Percentile £425 £400 £407
90th Percentile £713 £675 £700
UK excluding London median daily rate £525 £500 £475
% change year-on-year +5.00% +5.26% -13.64%
Number of hourly rates quoted 4 10 3
UK median hourly rate £50.00 £23.42 £20.50
Median hourly rate % change year-on-year +113.49% +14.24% -38.90%
10th Percentile £42.50 £16.64 £19.25
90th Percentile £140.98 £51.88 £92.35
UK excluding London median hourly rate - £19.34 £20.50
% change year-on-year - -5.66% -38.90%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 16,388 16,555 17,557
As % of all contract IT jobs advertised in the UK 16.47% 16.51% 18.18%
Number of daily rates quoted 10,564 11,053 11,315
UK median daily rate £469 £450 £425
Median daily rate % change year-on-year +4.11% +5.88% +6.25%
10th Percentile £275 £263 £250
90th Percentile £631 £620 £594
UK excluding London median daily rate £425 £400 £388
% change year-on-year +6.25% +3.23% -
Number of hourly rates quoted 352 397 404
UK median hourly rate £20.00 £19.50 £19.44
Median hourly rate % change year-on-year +2.56% +0.33% -7.45%
10th Percentile £12.00 £11.25 £12.00
90th Percentile £47.50 £47.50 £47.00
UK excluding London median hourly rate £19.57 £19.34 £20.00
% change year-on-year +1.20% -3.30% -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 16 January 2019.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 16 January 2019.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 16 January 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -21 1,377 £550 - 299
London -6 929 £575 +2.22% 204
UK excluding London -20 485 £525 +5.00% 115
South East -22 222 £525 +5.00% 39
North of England +17 95 £525 +7.69% 22
South West +5 66 £550 +15.79% 15
Yorkshire +14 53 £500 +17.65% 14
East of England -7 36 £550 +12.82% 7
North West -17 33 £550 - 7
Scotland +2 24 £500 +14.29% 15
Midlands -30 20 £438 -4.11% 12
East Midlands +10 18 £450 -1.37% 4
Wales +5 17 £498 - 3
North East -1 9 £500 +25.00% 1
Northern Ireland +28 5 £434 - 2
West Midlands -38 2 £425 -8.11% 8

For the 6 months to 16 January 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 932 (65.27%) Hadoop
2 927 (64.92%) Big Data
3 745 (52.17%) Python
4 643 (45.03%) Kafka
5 609 (42.65%) Java
6 521 (36.48%) Apache Hive
7 485 (33.96%) SQL
8 458 (32.07%) Agile Software Development
9 453 (31.72%) Scala
10 411 (28.78%) Data Science
11 407 (28.50%) Finance
12 404 (28.29%) Analytics
13 324 (22.69%) Amazon AWS
14 295 (20.66%) Apache
15 279 (19.54%) Machine Learning
16 260 (18.21%) HBase
17 255 (17.86%) ETL
18 251 (17.58%) Elasticsearch
19 247 (17.30%) NoSQL
20 231 (16.18%) Linux
21 228 (15.97%) Data Modelling
22 226 (15.83%) Apache Cassandra
23 211 (14.78%) Data Warehouse
24 209 (14.64%) Microsoft Azure
25 208 (14.57%) DevOps
26 198 (13.87%) Docker
27 195 (13.66%) TDD
28 182 (12.75%) Jenkins
29 171 (11.97%) Software Engineering
30 166 (11.62%) HDFS

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 295 (20.66%) Apache
2 251 (17.58%) Elasticsearch
3 54 (3.78%) Apache Pig
4 48 (3.36%) SAS
5 46 (3.22%) Apache Solr
6 38 (2.66%) Jupyter
7 34 (2.38%) Confluence
8 27 (1.89%) Apache Airflow
9 19 (1.33%) nginx
10 15 (1.05%) SharePoint
11 14 (0.98%) Oracle Coherence
12 9 (0.63%) Apache Qpid
12 9 (0.63%) Oracle SOA Suite
13 8 (0.56%) Tomcat
14 7 (0.49%) Cloud Foundry
15 6 (0.42%) Luigi
16 4 (0.28%) JBoss
17 3 (0.21%) GemFire
17 3 (0.21%) OpenStack
18 1 (0.070%) Apache ServiceMix
Applications
1 6 (0.42%) Microsoft Excel
2 5 (0.35%) MS Visio
3 4 (0.28%) Chatbot
4 2 (0.14%) GNU Octave
5 1 (0.070%) Microsoft PowerPoint
5 1 (0.070%) Microsoft Project
5 1 (0.070%) Spreadsheet
Business Applications
1 13 (0.91%) SAP Oil and Gas
2 3 (0.21%) SAP MM
2 3 (0.21%) SAP R/3
3 2 (0.14%) Dynamics NAV
3 2 (0.14%) Markit EDM
3 2 (0.14%) PolarLake
4 1 (0.070%) Dynamics AX
4 1 (0.070%) Dynamics CRM
4 1 (0.070%) Oracle Financials
4 1 (0.070%) Salesforce.com CRM
Cloud Services
1 324 (22.69%) Amazon AWS
2 209 (14.64%) Microsoft Azure
3 107 (7.49%) Google Cloud Platform
4 69 (4.83%) Amazon EMR
5 54 (3.78%) Amazon S3
6 42 (2.94%) Amazon Kinesis
7 37 (2.59%) PaaS
7 37 (2.59%) Serverless
8 34 (2.38%) GitHub
9 31 (2.17%) Azure Data Factory
10 23 (1.61%) Cloud Computing
11 22 (1.54%) Amazon EC2
12 21 (1.47%) SaaS
13 20 (1.40%) AWS Lambda
14 19 (1.33%) Data Lake Analytics
15 17 (1.19%) Azure Machine Learning
16 14 (0.98%) Heroku
17 10 (0.70%) IaaS
18 9 (0.63%) Azure Functions
19 8 (0.56%) Cortana Intelligence Suite
Communications & Networking
1 45 (3.15%) Kerberos
2 43 (3.01%) HTTP
3 42 (2.94%) TCP/IP
4 31 (2.17%) Firewall
5 30 (2.10%) Internet
6 26 (1.82%) DNS
7 23 (1.61%) SSL
8 22 (1.54%) SSH
9 15 (1.05%) VPN
10 14 (0.98%) FTP
11 11 (0.77%) DAB
11 11 (0.77%) HTTPS
12 3 (0.21%) DSL
12 3 (0.21%) ICMP
13 2 (0.14%) IPv6
14 1 (0.070%) AMQP
14 1 (0.070%) Broadband
14 1 (0.070%) Network Security
Database & Business Intelligence
1 932 (65.27%) Hadoop
2 927 (64.92%) Big Data
3 521 (36.48%) Apache Hive
4 260 (18.21%) HBase
5 247 (17.30%) NoSQL
6 226 (15.83%) Apache Cassandra
7 211 (14.78%) Data Warehouse
8 164 (11.48%) MongoDB
9 155 (10.85%) Impala
10 151 (10.57%) Relational Database
11 142 (9.94%) Data Lake
12 113 (7.91%) SQL Server
13 105 (7.35%) Sqoop
14 90 (6.30%) PostgreSQL
15 56 (3.92%) Flink
16 52 (3.64%) RDBMS
17 51 (3.57%) Enterprise Data Warehouse
18 50 (3.50%) Amazon Redshift
18 50 (3.50%) OLAP
19 49 (3.43%) Star Schema
Development Applications
1 182 (12.75%) Jenkins
2 162 (11.34%) Git (software)
3 63 (4.41%) JIRA
4 46 (3.22%) Maven
5 38 (2.66%) Subversion
6 32 (2.24%) Cucumber
7 29 (2.03%) Visual Studio Team System
8 28 (1.96%) JUnit
9 26 (1.82%) Atlassian Bamboo
9 26 (1.82%) GitLab
10 21 (1.47%) Gradle
11 17 (1.19%) Selenium
11 17 (1.19%) Visual Studio
12 16 (1.12%) gulp
13 15 (1.05%) Mercurial
14 13 (0.91%) Artifactory
15 12 (0.84%) FishEye
16 11 (0.77%) IntelliJ
16 11 (0.77%) JBehave
17 10 (0.70%) jMock
General
1 407 (28.50%) Finance
2 120 (8.40%) Banking
3 94 (6.58%) Front Office
4 66 (4.62%) Legal
5 58 (4.06%) Retail
6 28 (1.96%) Telecoms
7 27 (1.89%) Marketing
8 26 (1.82%) Investment Banking
9 13 (0.91%) Games
9 13 (0.91%) Publishing
10 6 (0.42%) Retail Banking
11 3 (0.21%) Advertising
11 3 (0.21%) Financial Institution
11 3 (0.21%) Manufacturing
12 2 (0.14%) Aerospace
12 2 (0.14%) Back Office
12 2 (0.14%) Pharmaceutical
13 1 (0.070%) Automotive
13 1 (0.070%) Electronics
13 1 (0.070%) Law
Job Titles
1 502 (35.15%) Developer
2 445 (31.16%) Data Engineer
3 233 (16.32%) Big Data Engineer
4 168 (11.76%) Java Developer
5 142 (9.94%) Big Data Developer
6 131 (9.17%) Apache Spark Developer
7 122 (8.54%) Data Scientist
8 116 (8.12%) Architect
9 98 (6.86%) Hadoop Engineer
10 88 (6.16%) Python Developer
11 81 (5.67%) Python Engineer
12 67 (4.69%) Senior Developer
13 66 (4.62%) Data Architect
14 61 (4.27%) Scala Developer
15 59 (4.13%) Java Engineer
16 57 (3.99%) DevOps Engineer
17 56 (3.92%) Software Engineer
18 50 (3.50%) Senior Data Engineer
19 49 (3.43%) Hadoop Developer
20 40 (2.80%) Big Data Architect
Libraries, Frameworks & Software Standards
1 643 (45.03%) Kafka
2 130 (9.10%) Spark Streaming
3 128 (8.96%) Spring
4 118 (8.26%) PySpark
5 97 (6.79%) Pandas
6 90 (6.30%) NumPy
7 81 (5.67%) REST
8 76 (5.32%) TensorFlow
9 63 (4.41%) JSON
10 60 (4.20%) Apache NiFi
10 60 (4.20%) RabbitMQ
11 56 (3.92%) Natural Language Processing
12 53 (3.71%) RESTful
13 52 (3.64%) Spring Boot
14 43 (3.01%) Elastic Stack
15 40 (2.80%) Web Services
16 38 (2.66%) scikit-learn
17 37 (2.59%) AngularJS
18 36 (2.52%) Akka
19 35 (2.45%) Theano
Miscellaneous
1 163 (11.41%) Algorithms
2 139 (9.73%) Greenfield Project
3 109 (7.63%) Analytical Skills
4 63 (4.41%) Foreign Exchange (FX)
5 58 (4.06%) Data Structures
6 45 (3.15%) Fintech
7 24 (1.68%) Market Risk
8 18 (1.26%) Enterprise Software
9 16 (1.12%) Virtual Team
10 14 (0.98%) Self-Motivation
11 12 (0.84%) Data Centre
12 11 (0.77%) Grid Computing
12 11 (0.77%) HoloLens
13 8 (0.56%) Derivative
13 8 (0.56%) Fixed Income
14 6 (0.42%) Replication
15 5 (0.35%) Equities
15 5 (0.35%) Internet of Things
15 5 (0.35%) Mobile App
15 5 (0.35%) User Experience
Operating Systems
1 231 (16.18%) Linux
2 113 (7.91%) Unix
3 18 (1.26%) Red Hat Enterprise Linux
4 12 (0.84%) Windows
5 5 (0.35%) Solaris
6 2 (0.14%) Android
6 2 (0.14%) Apple iOS
6 2 (0.14%) CentOS
6 2 (0.14%) DC/OS
6 2 (0.14%) Ubuntu
7 1 (0.070%) Mac OS X
7 1 (0.070%) Mesosphere
7 1 (0.070%) Oracle Linux
Processes & Methodologies
1 458 (32.07%) Agile Software Development
2 411 (28.78%) Data Science
3 404 (28.29%) Analytics
4 279 (19.54%) Machine Learning
5 255 (17.86%) ETL
6 228 (15.97%) Data Modelling
7 208 (14.57%) DevOps
8 195 (13.66%) TDD
9 171 (11.97%) Software Engineering
10 157 (10.99%) Business Intelligence
11 155 (10.85%) Open Source
12 143 (10.01%) Visualisation
13 134 (9.38%) Continuous Integration
14 127 (8.89%) BDD
15 121 (8.47%) Microservices
16 120 (8.40%) Computer Science
17 117 (8.19%) Scrum
18 116 (8.12%) Data Engineering
19 112 (7.84%) MapReduce
20 106 (7.42%) Data Analytics
Programming Languages
1 745 (52.17%) Python
2 609 (42.65%) Java
3 485 (33.96%) SQL
4 453 (31.72%) Scala
5 136 (9.52%) R
6 106 (7.42%) Java 8
7 62 (4.34%) Bash Shell
8 56 (3.92%) JavaScript
9 51 (3.57%) Shell Script
10 48 (3.36%) C#
11 42 (2.94%) MATLAB
12 33 (2.31%) C++
13 20 (1.40%) Ruby
14 18 (1.26%) Go
15 17 (1.19%) Perl
15 17 (1.19%) TypeScript
16 16 (1.12%) PL/SQL
17 14 (0.98%) ES6
17 14 (0.98%) Groovy
18 12 (0.84%) Cypher
Qualifications
1 105 (7.35%) Degree
2 65 (4.55%) Security Cleared
3 53 (3.71%) SC Cleared
4 51 (3.57%) PhD
5 44 (3.08%) Master's Degree
6 42 (2.94%) Computer Science Degree
7 18 (1.26%) Numerate Degree
8 13 (0.91%) BPSS Clearance
9 12 (0.84%) ISTQB
10 6 (0.42%) DV Cleared
11 4 (0.28%) ISTQB Advanced Certification
11 4 (0.28%) TOGAF Certification
12 3 (0.21%) AWS Certification
12 3 (0.21%) CFA
13 2 (0.14%) MBA
14 1 (0.070%) CCA
14 1 (0.070%) ISEB
14 1 (0.070%) MCSA
14 1 (0.070%) MCSE
14 1 (0.070%) Microsoft Certification
Quality Assurance & Compliance
1 76 (5.32%) Data Quality
2 33 (2.31%) QA
3 22 (1.54%) GDPR
4 9 (0.63%) ISO/IEC 27001
5 5 (0.35%) HL7
6 4 (0.28%) EMIR
6 4 (0.28%) MiFID
7 3 (0.21%) SLA
8 1 (0.070%) PMO
8 1 (0.070%) PSD2
System Software
1 198 (13.87%) Docker
2 166 (11.62%) HDFS
3 92 (6.44%) Apache Flume
4 49 (3.43%) Apache ZooKeeper
5 13 (0.91%) Active Directory
6 3 (0.21%) Virtual Machines
6 3 (0.21%) Virtual Servers
7 2 (0.14%) Tivoli Workload Scheduler
8 1 (0.070%) NFS
8 1 (0.070%) Oracle RAC
8 1 (0.070%) VMware ESXi
8 1 (0.070%) VMware Infrastructure
Systems Management
1 148 (10.36%) Kubernetes
2 129 (9.03%) Oozie
3 116 (8.12%) Hadoop YARN
4 107 (7.49%) Ansible
5 53 (3.71%) Terraform
6 32 (2.24%) Kibana
7 30 (2.10%) logstash
7 30 (2.10%) Puppet
8 26 (1.82%) Opscode Chef
9 19 (1.33%) Ambari
10 15 (1.05%) Docker Swarm
10 15 (1.05%) ITRS Geneos
11 12 (0.84%) Mesos
12 11 (0.77%) Control-M
13 9 (0.63%) Tivoli
14 7 (0.49%) Marathon
14 7 (0.49%) Single Sign-On
15 5 (0.35%) Prometheus
16 4 (0.28%) rkt
17 3 (0.21%) Sysdig
Vendors
1 163 (11.41%) Cloudera
2 125 (8.75%) Oracle
3 111 (7.77%) Hortonworks
4 90 (6.30%) Google
5 76 (5.32%) Splunk
6 67 (4.69%) Tableau
7 60 (4.20%) Microsoft
8 58 (4.06%) Talend
9 51 (3.57%) Informatica
10 46 (3.22%) Databricks
11 33 (2.31%) MapR
12 23 (1.61%) Alteryx
13 21 (1.47%) SAP
14 18 (1.26%) Teradata
15 17 (1.19%) Atlassian
16 16 (1.12%) Ab Initio
17 15 (1.05%) Red Hat
18 11 (0.77%) IBM
18 11 (0.77%) Sybase
19 9 (0.63%) Netezza