Period
to 18 December 2018

The following table provides summary statistics for contract job vacancies with a requirement for Spark Streaming skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark Streaming over the 6 months to 18 December 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark Streaming
UK
6 months to
18 Dec 2018
Same period 2017 Same period 2016
Rank 761 771 790
Rank change year-on-year +10 +19 +114
Contract jobs citing Spark Streaming 143 127 100
As % of all contract IT jobs advertised in the UK 0.14% 0.12% 0.098%
As % of the Libraries, Frameworks & Software Standards category 0.39% 0.38% 0.32%
Number of daily rates quoted 101 76 63
UK median daily rate £575 £520 £550
Median daily rate % change year-on-year +10.58% -5.45% +9.89%
10th Percentile £463 £422 £459
90th Percentile £775 £613 £745
UK excluding London median daily rate £550 £488 £550
% change year-on-year +12.82% -11.36% +9.89%
Number of hourly rates quoted 0 1 0
UK median hourly rate - £35.00 -
UK excluding London median hourly rate - £35.00 -

Spark Streaming is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 36,389 33,232 31,610
As % of all contract IT jobs advertised in the UK 34.71% 32.17% 31.04%
Number of daily rates quoted 25,739 23,498 21,807
UK median daily rate £475 £450 £425
Median daily rate % change year-on-year +5.56% +5.88% +3.03%
10th Percentile £325 £313 £300
90th Percentile £638 £625 £588
UK excluding London median daily rate £425 £400 £390
% change year-on-year +6.25% +2.56% +4.00%
Number of hourly rates quoted 314 362 358
UK median hourly rate £41.52 £40.00 £40.00
Median hourly rate % change year-on-year +3.79% - +6.67%
10th Percentile £19.00 £17.78 £14.56
90th Percentile £60.00 £52.45 £51.80
UK excluding London median hourly rate £41.53 £40.00 £39.60
% change year-on-year +3.83% +1.01% +5.60%

Spark Streaming
Job Vacancy Trend

Job postings citing Spark Streaming as a percentage of all IT jobs advertised.

Job vacancy trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark Streaming.

Contractor daily rate trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Spark Streaming over the 6 months to 18 December 2018.

Contractor daily rate histogram for Spark Streaming in the UK

Spark Streaming
Top 12 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark Streaming within the UK over the 6 months to 18 December 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +19 141 £575 +11.65% 8
UK excluding London -23 74 £550 +12.82% 6
London +35 68 £663 +26.19% 3
South East +14 54 £550 +10.00% 2
North of England -29 11 £525 +13.88% 1
North West - 7 £525 - 1
East Midlands - 4 £320 -
Midlands - 4 £320 - 1
North East - 4 £500 -
East of England -13 2 - -
Scotland - 2 £525 - 1
South West - 1 £550 - 1

For the 6 months to 18 December 2018, IT contractor jobs citing Spark Streaming also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Spark Streaming.

1 143 (100.00%) Apache Spark
2 116 (81.12%) Kafka
3 115 (80.42%) Hadoop
4 110 (76.92%) Big Data
5 94 (65.73%) Java
6 86 (60.14%) Apache Hive
7 79 (55.24%) HBase
8 76 (53.15%) Python
9 66 (46.15%) Agile Software Development
10 60 (41.96%) Hadoop YARN
11 59 (41.26%) Scala
12 53 (37.06%) Apache
13 46 (32.17%) ETL
14 44 (30.77%) SQL
15 42 (29.37%) TDD
16 41 (28.67%) Unix
17 40 (27.97%) HDFS
18 39 (27.27%) Linux
18 39 (27.27%) BDD
19 37 (25.87%) Jenkins
19 37 (25.87%) Apache Cassandra
20 35 (24.48%) RESTful
20 35 (24.48%) Finance
21 34 (23.78%) RabbitMQ
21 34 (23.78%) Cloudera
22 33 (23.08%) Apache ZooKeeper
23 32 (22.38%) Analytics
24 31 (21.68%) Spring
24 31 (21.68%) Open Source
25 30 (20.98%) MongoDB

Spark Streaming
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 143 (100.00%) Apache Spark
2 53 (37.06%) Apache
3 25 (17.48%) Elasticsearch
4 16 (11.19%) Apache Pig
5 8 (5.59%) Apache Solr
5 8 (5.59%) GemFire
6 2 (1.40%) Cloud Foundry
7 1 (0.70%) Apache Airflow
7 1 (0.70%) Luigi
Cloud Services
1 22 (15.38%) Amazon AWS
2 17 (11.89%) GitHub
3 13 (9.09%) Google Cloud Platform
4 10 (6.99%) Amazon Kinesis
5 5 (3.50%) Microsoft Azure
6 4 (2.80%) Amazon EMR
6 4 (2.80%) Amazon S3
6 4 (2.80%) Cloud Computing
6 4 (2.80%) PaaS
7 3 (2.10%) AWS Lambda
7 3 (2.10%) Serverless
8 2 (1.40%) SaaS
9 1 (0.70%) Amazon CloudWatch
9 1 (0.70%) Amazon EC2
9 1 (0.70%) Amazon SQS
Communications & Networking
1 20 (13.99%) SSH
2 19 (13.29%) Firewall
2 19 (13.29%) Internet
2 19 (13.29%) TCP/IP
3 6 (4.20%) HTTP
3 6 (4.20%) Kerberos
4 4 (2.80%) DNS
5 2 (1.40%) HTTPS
Database & Business Intelligence
1 115 (80.42%) Hadoop
2 110 (76.92%) Big Data
3 86 (60.14%) Apache Hive
4 79 (55.24%) HBase
5 37 (25.87%) Apache Cassandra
6 30 (20.98%) Flink
6 30 (20.98%) Impala
6 30 (20.98%) MongoDB
7 29 (20.28%) NoSQL
8 18 (12.59%) Sqoop
9 15 (10.49%) Presto
9 15 (10.49%) Relational Database
10 12 (8.39%) Spark SQL
11 8 (5.59%) OLTP
12 7 (4.90%) Data Lake
12 7 (4.90%) MySQL
12 7 (4.90%) SQL Server
13 6 (4.20%) Data Warehouse
14 5 (3.50%) DataStax
15 4 (2.80%) Enterprise Data Warehouse
Development Applications
1 37 (25.87%) Jenkins
2 18 (12.59%) Git (software)
3 12 (8.39%) JIRA
4 8 (5.59%) JUnit
5 7 (4.90%) Gatling
6 5 (3.50%) Travis CI
7 4 (2.80%) Eclipse
7 4 (2.80%) IntelliJ
8 3 (2.10%) Atlassian Bamboo
8 3 (2.10%) EasyMock
8 3 (2.10%) Mockito
8 3 (2.10%) TeamCity
8 3 (2.10%) TestNG
9 2 (1.40%) GitLab
9 2 (1.40%) Gradle
9 2 (1.40%) Maven
10 1 (0.70%) Cucumber
10 1 (0.70%) Visual Studio Team System
General
1 35 (24.48%) Finance
2 21 (14.69%) Legal
3 19 (13.29%) Banking
4 12 (8.39%) Publishing
5 6 (4.20%) Telecoms
6 5 (3.50%) Investment Banking
6 5 (3.50%) Retail Banking
7 3 (2.10%) Games
Job Titles
1 76 (53.15%) Developer
2 52 (36.36%) Big Data Developer
3 42 (29.37%) Data Engineer
4 27 (18.88%) Big Data Engineer
4 27 (18.88%) Java Developer
5 17 (11.89%) Apache Spark Developer
5 17 (11.89%) Architect
6 16 (11.19%) Hadoop Developer
7 15 (10.49%) Analytics Developer
7 15 (10.49%) Big Data Architect
8 13 (9.09%) Data Architect
8 13 (9.09%) Python Engineer
8 13 (9.09%) Software Developer
9 8 (5.59%) Java Hadoop Developer
10 5 (3.50%) Software Engineer
11 4 (2.80%) Big Data Software Engineer
11 4 (2.80%) Data Analytics Engineer
11 4 (2.80%) Senior Developer
12 3 (2.10%) Cloud Engineer
12 3 (2.10%) Hadoop Engineer
Libraries, Frameworks & Software Standards
1 116 (81.12%) Kafka
2 35 (24.48%) RESTful
3 34 (23.78%) RabbitMQ
4 31 (21.68%) Spring
5 22 (15.38%) J2EE
6 21 (14.69%) JSON
6 21 (14.69%) Middleware
7 15 (10.49%) Spring Boot
8 10 (6.99%) Elastic Stack
9 7 (4.90%) Java Concurrency
9 7 (4.90%) Lambda Expressions
10 6 (4.20%) ActiveMQ
11 4 (2.80%) Akka
11 4 (2.80%) WebSockets
12 3 (2.10%) Apache Avro
12 3 (2.10%) WireMock
13 2 (1.40%) Play Framework
13 2 (1.40%) PySpark
13 2 (1.40%) spray
13 2 (1.40%) Spring Integration
Miscellaneous
1 24 (16.78%) Analytical Skills
2 12 (8.39%) Algorithms
3 7 (4.90%) Greenfield Project
4 5 (3.50%) Cyber Defence
5 4 (2.80%) Data Centre
5 4 (2.80%) Data Structures
6 2 (1.40%) Cyberattack
6 2 (1.40%) Fintech
6 2 (1.40%) Linux Command Line
6 2 (1.40%) Self-Motivation
6 2 (1.40%) Virtual Team
Operating Systems
1 41 (28.67%) Unix
2 39 (27.27%) Linux
3 1 (0.70%) Windows
Processes & Methodologies
1 66 (46.15%) Agile Software Development
2 46 (32.17%) ETL
3 42 (29.37%) TDD
4 39 (27.27%) BDD
5 32 (22.38%) Analytics
6 31 (21.68%) Open Source
7 26 (18.18%) Data Modelling
8 23 (16.08%) Data Ingestion
8 23 (16.08%) Scrum
9 20 (13.99%) Data Analytics
9 20 (13.99%) Performance Tuning
9 20 (13.99%) Unit Testing
9 20 (13.99%) Web Development
10 19 (13.29%) Big Data Analytics
10 19 (13.29%) Business Intelligence
10 19 (13.29%) Data Protection
10 19 (13.29%) Kalman Filter
10 19 (13.29%) MapReduce
10 19 (13.29%) Network Architecture
11 16 (11.19%) Deployment Automation
Programming Languages
1 94 (65.73%) Java
2 76 (53.15%) Python
3 59 (41.26%) Scala
4 44 (30.77%) SQL
5 7 (4.90%) Shell Script
6 6 (4.20%) Bash Shell
6 6 (4.20%) Java 8
6 6 (4.20%) R
7 4 (2.80%) C++
8 3 (2.10%) PL/SQL
9 2 (1.40%) Cypher
10 1 (0.70%) C
10 1 (0.70%) C#
10 1 (0.70%) Perl
Qualifications
1 4 (2.80%) ISTQB
1 4 (2.80%) ISTQB Advanced Certification
2 2 (1.40%) Computer Science Degree
2 2 (1.40%) Degree
Quality Assurance & Compliance
1 7 (4.90%) QA
System Software
1 40 (27.97%) HDFS
2 33 (23.08%) Apache ZooKeeper
3 14 (9.79%) Apache Flume
3 14 (9.79%) Docker
Systems Management
1 60 (41.96%) Hadoop YARN
2 17 (11.89%) Ansible
3 10 (6.99%) Kubernetes
4 9 (6.29%) Kibana
4 9 (6.29%) logstash
5 5 (3.50%) Oozie
6 4 (2.80%) Control-M
6 4 (2.80%) Opscode Chef
6 4 (2.80%) Puppet
6 4 (2.80%) Tivoli
7 2 (1.40%) Prometheus
8 1 (0.70%) Terraform
Vendors
1 34 (23.78%) Cloudera
2 11 (7.69%) Google
2 11 (7.69%) Hortonworks
3 8 (5.59%) MapR
4 7 (4.90%) Oracle
5 4 (2.80%) SaltStack
5 4 (2.80%) Tableau
6 1 (0.70%) Databricks
6 1 (0.70%) Solace
6 1 (0.70%) Talend