Period
to 17 November 2018

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 17 November 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
17 Nov 2018
Same period 2017 Same period 2016
Rank 200 182 240
Rank change year-on-year -18 +58 +257
Contract jobs citing Apache Spark 1,437 1,491 1,151
As % of all contract IT jobs advertised in the UK 1.39% 1.45% 1.09%
As % of the Application Platforms category 8.28% 8.70% 5.85%
Number of daily rates quoted 998 1,089 791
UK median daily rate £550 £550 £550
10th Percentile £425 £400 £400
90th Percentile £713 £675 £688
UK excluding London median daily rate £525 £475 £450
% change year-on-year +10.53% +5.56% -10.18%
Number of hourly rates quoted 8 10 3
UK median hourly rate £33.75 £23.42 £20.50
Median hourly rate % change year-on-year +44.11% +14.24% -38.90%
10th Percentile £16.48 £16.64 £19.25
90th Percentile £98.53 £47.50 £92.35
UK excluding London median hourly rate £22.00 £23.42 £20.50
% change year-on-year -6.06% +14.24% -38.90%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 17,357 17,136 19,685
As % of all contract IT jobs advertised in the UK 16.83% 16.66% 18.57%
Number of daily rates quoted 11,278 11,238 12,776
UK median daily rate £450 £450 £420
Median daily rate % change year-on-year - +7.14% +5.00%
10th Percentile £275 £263 £250
90th Percentile £625 £606 £588
UK excluding London median daily rate £425 £400 £388
% change year-on-year +6.25% +3.23% +3.33%
Number of hourly rates quoted 397 438 495
UK median hourly rate £22.00 £20.22 £20.00
Median hourly rate % change year-on-year +8.82% +1.09% -
10th Percentile £12.25 £11.00 £12.20
90th Percentile £47.50 £47.83 £48.75
UK excluding London median hourly rate £22.00 £20.00 £20.73
% change year-on-year +10.00% -3.52% +8.05%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark.

Contractor daily rate trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Apache Spark over the 6 months to 17 November 2018.

Contractor daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark.

Contractor hourly rate trend for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Histogram

The hourly rate distribution of IT jobs citing Apache Spark over the 6 months to 17 November 2018.

Contractor hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 17 November 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -17 1,376 £550 - 311
London -13 926 £575 +4.55% 192
UK excluding London +5 500 £525 +10.53% 133
South East -16 222 £525 +5.00% 41
North of England -20 90 £525 +14.13% 32
South West +15 63 £525 +23.53% 24
Yorkshire -8 49 £488 +11.43% 10
East of England -14 45 £550 +12.82% 9
North West -9 40 £534 +5.43% 17
Scotland +9 33 £425 +6.25% 5
Midlands -36 24 £425 -5.56% 13
East Midlands +25 20 £463 -6.09% 8
Wales -5 13 £498 - 4
Northern Ireland +39 9 £432 - 5
West Midlands -55 4 £375 -16.67% 5
North East -10 1 £413 +3.13% 5

For the 6 months to 17 November 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 992 (69.03%) Hadoop
2 954 (66.39%) Big Data
3 720 (50.10%) Python
4 674 (46.90%) Kafka
5 652 (45.37%) Java
6 496 (34.52%) Agile Software Development
7 494 (34.38%) Scala
7 494 (34.38%) Apache Hive
8 484 (33.68%) SQL
9 417 (29.02%) Finance
10 390 (27.14%) Analytics
11 367 (25.54%) Data Science
12 321 (22.34%) Amazon AWS
13 297 (20.67%) Apache
14 291 (20.25%) HBase
15 278 (19.35%) NoSQL
16 262 (18.23%) Linux
17 247 (17.19%) ETL
18 238 (16.56%) Machine Learning
19 232 (16.14%) TDD
20 227 (15.80%) Elasticsearch
21 223 (15.52%) Apache Cassandra
22 209 (14.54%) DevOps
23 208 (14.47%) HDFS
24 204 (14.20%) Jenkins
25 192 (13.36%) Git (software)
26 191 (13.29%) Docker
27 183 (12.73%) Data Modelling
27 183 (12.73%) Business Intelligence
28 177 (12.32%) Relational Database

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 297 (20.67%) Apache
2 227 (15.80%) Elasticsearch
3 71 (4.94%) Apache Pig
4 62 (4.31%) Apache Solr
5 61 (4.24%) SAS
6 36 (2.51%) Confluence
7 30 (2.09%) Apache Airflow
8 21 (1.46%) Jupyter
9 20 (1.39%) nginx
10 19 (1.32%) SharePoint
11 13 (0.90%) Oracle Coherence
12 9 (0.63%) GemFire
12 9 (0.63%) Luigi
12 9 (0.63%) Tomcat
13 8 (0.56%) Oracle SOA Suite
14 7 (0.49%) Cloud Foundry
15 5 (0.35%) Apache Qpid
16 3 (0.21%) BizTalk Server
16 3 (0.21%) JBoss
16 3 (0.21%) Oracle Workflow
Applications
1 4 (0.28%) Microsoft Excel
2 2 (0.14%) Chatbot
2 2 (0.14%) GNU Octave
2 2 (0.14%) Microsoft PowerPoint
2 2 (0.14%) MS Visio
3 1 (0.070%) Microsoft Office
3 1 (0.070%) Microsoft Project
3 1 (0.070%) Shockwave
3 1 (0.070%) Spreadsheet
3 1 (0.070%) SPSS
3 1 (0.070%) Stata
Business Applications
1 12 (0.84%) SAP Oil and Gas
2 3 (0.21%) Dynamics AX
2 3 (0.21%) Dynamics CRM
2 3 (0.21%) SAP MM
2 3 (0.21%) SAP R/3
3 2 (0.14%) Markit EDM
3 2 (0.14%) PolarLake
4 1 (0.070%) Oracle Financials
4 1 (0.070%) Remedy ITSM
4 1 (0.070%) Salesforce.com CRM
Cloud Services
1 321 (22.34%) Amazon AWS
2 174 (12.11%) Microsoft Azure
3 88 (6.12%) Google Cloud Platform
4 76 (5.29%) Amazon EMR
5 55 (3.83%) Amazon Kinesis
6 53 (3.69%) Amazon S3
7 38 (2.64%) GitHub
8 31 (2.16%) Amazon EC2
9 27 (1.88%) PaaS
10 25 (1.74%) Azure Data Factory
11 23 (1.60%) Cloud Computing
12 21 (1.46%) Serverless
13 18 (1.25%) AWS Lambda
14 15 (1.04%) Heroku
15 14 (0.97%) SaaS
16 12 (0.84%) Data Lake Analytics
17 9 (0.63%) Azure Machine Learning
18 8 (0.56%) IaaS
19 7 (0.49%) OpenShift
20 6 (0.42%) AWS CloudFormation
Communications & Networking
1 56 (3.90%) Kerberos
2 47 (3.27%) HTTP
3 41 (2.85%) TCP/IP
4 28 (1.95%) SSL
5 27 (1.88%) DNS
5 27 (1.88%) Firewall
6 26 (1.81%) Internet
7 20 (1.39%) SSH
8 16 (1.11%) FTP
9 14 (0.97%) VPN
10 11 (0.77%) HTTPS
11 10 (0.70%) DAB
12 4 (0.28%) SNA
13 3 (0.21%) DSL
13 3 (0.21%) ICMP
14 2 (0.14%) IPv6
14 2 (0.14%) RTMP
15 1 (0.070%) Broadband
15 1 (0.070%) DHCP
15 1 (0.070%) Intranet
Database & Business Intelligence
1 992 (69.03%) Hadoop
2 954 (66.39%) Big Data
3 494 (34.38%) Apache Hive
4 291 (20.25%) HBase
5 278 (19.35%) NoSQL
6 223 (15.52%) Apache Cassandra
7 177 (12.32%) Relational Database
8 175 (12.18%) Data Warehouse
9 173 (12.04%) MongoDB
10 159 (11.06%) Impala
11 137 (9.53%) SQL Server
12 120 (8.35%) Data Lake
13 105 (7.31%) PostgreSQL
14 100 (6.96%) Sqoop
15 77 (5.36%) Amazon Redshift
16 72 (5.01%) RDBMS
17 56 (3.90%) Power BI
18 53 (3.69%) MySQL
19 48 (3.34%) Flink
20 47 (3.27%) Data Mining
Development Applications
1 204 (14.20%) Jenkins
2 192 (13.36%) Git (software)
3 68 (4.73%) JIRA
4 45 (3.13%) Maven
5 40 (2.78%) Subversion
6 38 (2.64%) Visual Studio Team System
7 37 (2.57%) Cucumber
7 37 (2.57%) GitLab
8 26 (1.81%) JUnit
9 25 (1.74%) IntelliJ
10 24 (1.67%) Gradle
11 22 (1.53%) Atlassian Bamboo
12 19 (1.32%) Selenium
13 16 (1.11%) gulp
13 16 (1.11%) Mercurial
14 14 (0.97%) Artifactory
14 14 (0.97%) Eclipse
15 13 (0.90%) Travis CI
16 11 (0.77%) JBehave
16 11 (0.77%) Visual Studio
General
1 417 (29.02%) Finance
2 125 (8.70%) Banking
3 73 (5.08%) Front Office
4 53 (3.69%) Legal
5 52 (3.62%) Retail
6 32 (2.23%) Investment Banking
7 27 (1.88%) Telecoms
8 23 (1.60%) Marketing
9 18 (1.25%) Games
10 16 (1.11%) Publishing
11 11 (0.77%) Retail Banking
12 6 (0.42%) Advertising
13 5 (0.35%) Manufacturing
13 5 (0.35%) Prime Brokerage
14 3 (0.21%) Financial Institution
15 2 (0.14%) Back Office
15 2 (0.14%) Electronics
15 2 (0.14%) Law
15 2 (0.14%) Pharmaceutical
16 1 (0.070%) Automotive
Job Titles
1 535 (37.23%) Developer
2 433 (30.13%) Data Engineer
3 246 (17.12%) Big Data Engineer
4 180 (12.53%) Big Data Developer
4 180 (12.53%) Java Developer
5 127 (8.84%) Apache Spark Developer
6 120 (8.35%) Architect
7 105 (7.31%) Hadoop Engineer
8 89 (6.19%) Data Scientist
9 87 (6.05%) Python Developer
10 80 (5.57%) Data Architect
11 74 (5.15%) Python Engineer
12 70 (4.87%) Senior Developer
13 69 (4.80%) Hadoop Developer
14 68 (4.73%) Senior Data Engineer
15 62 (4.31%) Scala Developer
16 59 (4.11%) DevOps Engineer
16 59 (4.11%) Software Engineer
17 55 (3.83%) Big Data Architect
18 46 (3.20%) AWS Engineer
Libraries, Frameworks & Software Standards
1 674 (46.90%) Kafka
2 146 (10.16%) Spark Streaming
3 140 (9.74%) Spring
4 91 (6.33%) REST
5 85 (5.92%) RabbitMQ
6 84 (5.85%) JSON
7 77 (5.36%) Spring Boot
8 75 (5.22%) Pandas
8 75 (5.22%) RESTful
9 74 (5.15%) PySpark
10 61 (4.24%) NumPy
11 56 (3.90%) AngularJS
12 55 (3.83%) TensorFlow
13 50 (3.48%) Natural Language Processing
14 49 (3.41%) J2EE
15 46 (3.20%) Elastic Stack
16 43 (2.99%) Akka
16 43 (2.99%) HTML
17 42 (2.92%) .NET
18 40 (2.78%) Web Services
Miscellaneous
1 154 (10.72%) Algorithms
2 120 (8.35%) Greenfield Project
3 119 (8.28%) Computer Science
4 110 (7.65%) Analytical Skills
5 47 (3.27%) Fintech
5 47 (3.27%) Foreign Exchange (FX)
6 45 (3.13%) Data Structures
7 28 (1.95%) Enterprise Software
8 21 (1.46%) Self-Motivation
9 17 (1.18%) Fixed Income
9 17 (1.18%) Virtual Team
10 16 (1.11%) HoloLens
11 15 (1.04%) Derivative
11 15 (1.04%) Market Risk
12 8 (0.56%) Grid Computing
13 7 (0.49%) Equities
14 6 (0.42%) Data Centre
14 6 (0.42%) Linux Command Line
14 6 (0.42%) Management Information System
14 6 (0.42%) Replication
Operating Systems
1 262 (18.23%) Linux
2 128 (8.91%) Unix
3 19 (1.32%) Red Hat Enterprise Linux
4 11 (0.77%) Windows
5 5 (0.35%) Ubuntu
6 3 (0.21%) DC/OS
7 2 (0.14%) Apple iOS
7 2 (0.14%) CentOS
8 1 (0.070%) Android
8 1 (0.070%) Mac OS X
8 1 (0.070%) Mesosphere
8 1 (0.070%) Oracle Linux
Processes & Methodologies
1 496 (34.52%) Agile Software Development
2 390 (27.14%) Analytics
3 367 (25.54%) Data Science
4 247 (17.19%) ETL
5 238 (16.56%) Machine Learning
6 232 (16.14%) TDD
7 209 (14.54%) DevOps
8 183 (12.73%) Business Intelligence
8 183 (12.73%) Data Modelling
9 163 (11.34%) Open Source
9 163 (11.34%) Scrum
10 148 (10.30%) Visualisation
11 146 (10.16%) BDD
11 146 (10.16%) Software Engineering
12 143 (9.95%) Continuous Integration
13 114 (7.93%) Microservices
14 113 (7.86%) MapReduce
15 110 (7.65%) Data Analytics
16 107 (7.45%) Data Engineering
17 92 (6.40%) Problem-Solving
Programming Languages
1 720 (50.10%) Python
2 652 (45.37%) Java
3 494 (34.38%) Scala
4 484 (33.68%) SQL
5 141 (9.81%) R
6 127 (8.84%) Java 8
7 77 (5.36%) JavaScript
8 53 (3.69%) Bash Shell
9 51 (3.55%) Shell Script
10 49 (3.41%) C#
11 48 (3.34%) C++
12 44 (3.06%) MATLAB
13 28 (1.95%) TypeScript
14 25 (1.74%) Ruby
15 20 (1.39%) Perl
16 17 (1.18%) ES6
16 17 (1.18%) PL/SQL
17 16 (1.11%) C
17 16 (1.11%) Groovy
18 12 (0.84%) Go
Qualifications
1 111 (7.72%) Degree
2 70 (4.87%) Security Cleared
3 55 (3.83%) SC Cleared
4 49 (3.41%) Computer Science Degree
5 33 (2.30%) PhD
6 26 (1.81%) Master's Degree
7 13 (0.90%) ISTQB
8 12 (0.84%) Numerate Degree
9 9 (0.63%) BPSS Clearance
10 8 (0.56%) DV Cleared
11 6 (0.42%) Microsoft Certification
12 4 (0.28%) ISTQB Advanced Certification
13 3 (0.21%) CCA
13 3 (0.21%) ISEB
14 2 (0.14%) AWS Certification
14 2 (0.14%) Sun Certification
15 1 (0.070%) CISSP
15 1 (0.070%) CREST Certified
15 1 (0.070%) DBS Check
15 1 (0.070%) MCSE
Quality Assurance & Compliance
1 65 (4.52%) Data Quality
2 31 (2.16%) QA
3 26 (1.81%) GDPR
4 8 (0.56%) ISO/IEC 27001
5 4 (0.28%) EMIR
5 4 (0.28%) MiFID
5 4 (0.28%) SLA
6 2 (0.14%) HL7
6 2 (0.14%) PMO
7 1 (0.070%) PSD2
System Software
1 208 (14.47%) HDFS
2 191 (13.29%) Docker
3 86 (5.98%) Apache Flume
4 63 (4.38%) Apache ZooKeeper
5 18 (1.25%) Active Directory
6 3 (0.21%) Virtual Servers
7 2 (0.14%) Tivoli Workload Scheduler
8 1 (0.070%) VMware ESXi
8 1 (0.070%) VMware Infrastructure
Systems Management
1 138 (9.60%) Kubernetes
2 137 (9.53%) Oozie
3 126 (8.77%) Hadoop YARN
4 108 (7.52%) Ansible
5 60 (4.18%) Terraform
6 40 (2.78%) Puppet
7 32 (2.23%) Kibana
8 29 (2.02%) logstash
9 27 (1.88%) Opscode Chef
10 23 (1.60%) Ambari
11 17 (1.18%) Docker Swarm
12 16 (1.11%) ITRS Geneos
13 14 (0.97%) Mesos
14 11 (0.77%) Control-M
15 9 (0.63%) Marathon
16 7 (0.49%) Single Sign-On
16 7 (0.49%) Tivoli
17 6 (0.42%) Prometheus
18 5 (0.35%) Consul
18 5 (0.35%) Salt
Vendors
1 143 (9.95%) Oracle
2 135 (9.39%) Cloudera
3 98 (6.82%) Hortonworks
4 80 (5.57%) Google
5 70 (4.87%) Tableau
6 61 (4.24%) Microsoft
6 61 (4.24%) Splunk
7 60 (4.18%) Informatica
8 49 (3.41%) Talend
9 33 (2.30%) MapR
10 28 (1.95%) SAP
11 27 (1.88%) Databricks
12 26 (1.81%) Alteryx
13 20 (1.39%) Ab Initio
13 20 (1.39%) Teradata
14 15 (1.04%) Red Hat
15 14 (0.97%) IBM
16 13 (0.90%) Atlassian
16 13 (0.90%) Sybase
17 7 (0.49%) Cognos