Period
to 21 June 2018

The following table provides summary statistics for contract job vacancies with a requirement for Spark Streaming skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark Streaming over the 6 months to 21 June 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark Streaming
UK
6 months to
21 Jun 2018
Same period 2017 Same period 2016
Rank 715 795 877
Rank change year-on-year +80 +82 -15
Contract jobs citing Spark Streaming 164 51 22
As % of all contract IT jobs advertised in the UK 0.16% 0.056% 0.021%
As % of the Libraries, Frameworks & Software Standards category 0.49% 0.17% 0.072%
Number of daily rates quoted 112 38 9
UK median daily rate £525 £525 £575
Median daily rate % change year-on-year - -8.70% -
10th Percentile £425 £375 £490
90th Percentile £650 £663 £745
UK excluding London median daily rate £525 £425 £575
% change year-on-year +23.53% -26.09% -
Number of hourly rates quoted 0 0 0
UK median hourly rate - - -
UK excluding London median hourly rate - - -

Spark Streaming is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 33,291 29,223 30,445
As % of all contract IT jobs advertised in the UK 33.28% 32.06% 28.53%
Number of daily rates quoted 23,401 20,212 21,268
UK median daily rate £470 £440 £425
Median daily rate % change year-on-year +6.82% +3.53% +6.25%
10th Percentile £313 £300 £288
90th Percentile £625 £613 £600
UK excluding London median daily rate £415 £400 £375
% change year-on-year +3.75% +6.67% +3.45%
Number of hourly rates quoted 347 375 419
UK median hourly rate £42.27 £40.00 £38.00
Median hourly rate % change year-on-year +5.66% +5.26% +8.57%
10th Percentile £15.15 £15.55 £13.50
90th Percentile £65.90 £53.75 £57.00
UK excluding London median hourly rate £40.25 £40.00 £38.00
% change year-on-year +0.63% +5.26% +8.57%

Spark Streaming
Job Vacancy Trend

Job postings citing Spark Streaming as a percentage of all IT jobs advertised.

Job vacancy trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark Streaming.

Contractor daily rate trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Spark Streaming over the 6 months to 21 June 2018.

Contractor daily rate histogram for Spark Streaming in the UK

Spark Streaming
Top 11 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark Streaming within the UK over the 6 months to 21 June 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +60 162 £525 - 12
UK excluding London +76 93 £525 +23.53% 4
London -2 70 £550 - 9
South East +30 58 £525 +23.53% 3
North of England - 18 £500 -
Yorkshire - 17 £500 -
Midlands - 16 £500 -
East Midlands - 15 £500 -
North West - 1 £625 -
West Midlands - 1 £500 -
Scotland - 1 £338 - 1

For the 6 months to 21 June 2018, IT contractor jobs citing Spark Streaming also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Spark Streaming.

1 162 (98.78%) Apache Spark
2 135 (82.32%) Big Data
3 128 (78.05%) Kafka
4 112 (68.29%) Hadoop
5 96 (58.54%) Scala
6 93 (56.71%) Agile Software Development
7 90 (54.88%) Java
8 73 (44.51%) Apache Hive
9 65 (39.63%) Apache
10 60 (36.59%) Linux
11 57 (34.76%) RESTful
12 50 (30.49%) Finance
13 49 (29.88%) RabbitMQ
13 49 (29.88%) Impala
14 48 (29.27%) Hadoop YARN
14 48 (29.27%) Apache ZooKeeper
14 48 (29.27%) Cloudera
15 47 (28.66%) Jenkins
15 47 (28.66%) HBase
16 43 (26.22%) Spring
17 42 (25.61%) NoSQL
18 41 (25.00%) TDD
19 40 (24.39%) Microservices
20 38 (23.17%) Sqoop
21 37 (22.56%) Python
21 37 (22.56%) Unix
22 31 (18.90%) Continuous Integration
22 31 (18.90%) Hortonworks
22 31 (18.90%) Apache Cassandra
23 30 (18.29%) SQL Server

Spark Streaming
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 162 (98.78%) Apache Spark
2 65 (39.63%) Apache
3 13 (7.93%) Elasticsearch
4 10 (6.10%) Apache Airflow
4 10 (6.10%) GemFire
5 9 (5.49%) Apache Pig
6 4 (2.44%) Apache Solr
7 1 (0.61%) Cloud Foundry
7 1 (0.61%) Luigi
Cloud Services
1 18 (10.98%) Amazon AWS
2 15 (9.15%) Google Cloud Platform
3 10 (6.10%) GitHub
4 9 (5.49%) Amazon EMR
5 7 (4.27%) Microsoft Azure
6 2 (1.22%) Amazon Kinesis
6 2 (1.22%) PaaS
7 1 (0.61%) Amazon EC2
7 1 (0.61%) Amazon S3
7 1 (0.61%) IaaS
7 1 (0.61%) SaaS
Communications & Networking
1 20 (12.20%) Kerberos
2 15 (9.15%) HTTP
3 8 (4.88%) DSL
4 2 (1.22%) DNS
4 2 (1.22%) SSL
Database & Business Intelligence
1 135 (82.32%) Big Data
2 112 (68.29%) Hadoop
3 73 (44.51%) Apache Hive
4 49 (29.88%) Impala
5 47 (28.66%) HBase
6 42 (25.61%) NoSQL
7 38 (23.17%) Sqoop
8 31 (18.90%) Apache Cassandra
9 30 (18.29%) SQL Server
10 27 (16.46%) Data Warehouse
11 26 (15.85%) Cloudera CDH
12 21 (12.80%) MongoDB
13 15 (9.15%) Relational Database
14 11 (6.71%) MySQL
15 10 (6.10%) Apache Kudu
15 10 (6.10%) Data Lake
15 10 (6.10%) Spark SQL
16 8 (4.88%) Oracle Database
16 8 (4.88%) Oracle Database 12c
16 8 (4.88%) Pentaho
Development Applications
1 47 (28.66%) Jenkins
2 28 (17.07%) Git (software)
3 20 (12.20%) Team Foundation Server
4 13 (7.93%) JIRA
5 8 (4.88%) Oracle APEX
6 3 (1.83%) Visual Studio Team System
7 2 (1.22%) Maven
8 1 (0.61%) Appium
8 1 (0.61%) Cucumber
8 1 (0.61%) Eclipse
8 1 (0.61%) Gatling
8 1 (0.61%) GitLab
8 1 (0.61%) Gradle
8 1 (0.61%) IntelliJ
8 1 (0.61%) JUnit
8 1 (0.61%) Selenium
8 1 (0.61%) SonarQube
8 1 (0.61%) Sonatype Nexus
8 1 (0.61%) TeamCity
8 1 (0.61%) Travis CI
General
1 50 (30.49%) Finance
2 8 (4.88%) Games
2 8 (4.88%) Telecoms
3 7 (4.27%) Banking
4 5 (3.05%) Automotive
5 2 (1.22%) Retail Banking
6 1 (0.61%) French Language
6 1 (0.61%) Investment Banking
6 1 (0.61%) Publishing
Job Titles
1 118 (71.95%) Developer
2 67 (40.85%) Big Data Developer
3 44 (26.83%) Java Developer
4 41 (25.00%) Apache Spark Developer
5 31 (18.90%) Data Engineer
6 18 (10.98%) Big Data Engineer
7 16 (9.76%) Scala Developer
8 15 (9.15%) Hadoop Developer
9 11 (6.71%) Java Hadoop Developer
10 10 (6.10%) Architect
10 10 (6.10%) Consultant
10 10 (6.10%) Data Architect
11 9 (5.49%) Big Data Architect
12 8 (4.88%) Big Data Consultant
12 8 (4.88%) Business Developer
12 8 (4.88%) Business Intelligence Developer
12 8 (4.88%) Data Consultant
12 8 (4.88%) Python Developer
13 5 (3.05%) AWS Engineer
13 5 (3.05%) DevOps Engineer
Libraries, Frameworks & Software Standards
1 128 (78.05%) Kafka
2 57 (34.76%) RESTful
3 49 (29.88%) RabbitMQ
4 43 (26.22%) Spring
5 29 (17.68%) Spring Boot
6 23 (14.02%) J2EE
7 22 (13.41%) JSON
7 22 (13.41%) Middleware
8 19 (11.59%) Spring Integration
8 19 (11.59%) Web Services
9 15 (9.15%) Akka
9 15 (9.15%) Play Framework
9 15 (9.15%) spray
10 4 (2.44%) CSS
10 4 (2.44%) HTML
11 3 (1.83%) Elastic Stack
11 3 (1.83%) Node.js
12 2 (1.22%) Natural Language Processing
12 2 (1.22%) TensorFlow
12 2 (1.22%) Vue (Vue.js)
Miscellaneous
1 23 (14.02%) Greenfield Project
2 5 (3.05%) Analytical Skills
3 3 (1.83%) Self-Motivation
4 2 (1.22%) Algorithms
4 2 (1.22%) Data Structures
4 2 (1.22%) Hedge funds
5 1 (0.61%) Computer Science
5 1 (0.61%) Data Centre
5 1 (0.61%) Fintech
5 1 (0.61%) Linux Command Line
5 1 (0.61%) Taxonomies
5 1 (0.61%) Virtual Team
Operating Systems
1 60 (36.59%) Linux
2 37 (22.56%) Unix
3 1 (0.61%) CentOS
Processes & Methodologies
1 93 (56.71%) Agile Software Development
2 41 (25.00%) TDD
3 40 (24.39%) Microservices
4 31 (18.90%) Continuous Integration
5 25 (15.24%) Analytics
5 25 (15.24%) BDD
5 25 (15.24%) DevOps
5 25 (15.24%) ETL
6 23 (14.02%) MapReduce
7 21 (12.80%) Data Modelling
8 20 (12.20%) Machine Learning
9 19 (11.59%) Business Intelligence
10 18 (10.98%) Decision-Making
10 18 (10.98%) Kalman Filter
11 15 (9.15%) Coaching
11 15 (9.15%) Pair Programming
12 11 (6.71%) Unit Testing
13 10 (6.10%) System Testing
14 7 (4.27%) Refactoring
14 7 (4.27%) Reverse Engineering
Programming Languages
1 96 (58.54%) Scala
2 90 (54.88%) Java
3 37 (22.56%) Python
4 30 (18.29%) SQL
5 16 (9.76%) Shell Script
6 8 (4.88%) HiveQL
7 4 (2.44%) C#
7 4 (2.44%) JavaScript
7 4 (2.44%) PHP
7 4 (2.44%) Ruby
8 3 (1.83%) Bash Shell
8 3 (1.83%) C++
9 2 (1.22%) Java 8
10 1 (0.61%) Groovy
Qualifications
1 19 (11.59%) Degree
2 8 (4.88%) Security Cleared
3 7 (4.27%) SC Cleared
4 1 (0.61%) Computer Science Degree
Quality Assurance & Compliance
1 9 (5.49%) GCP
2 1 (0.61%) Data Quality
2 1 (0.61%) GDPR
2 1 (0.61%) PSD2
2 1 (0.61%) QA
System Software
1 48 (29.27%) Apache ZooKeeper
2 29 (17.68%) HDFS
3 19 (11.59%) Docker
4 16 (9.76%) Apache Flume
Systems Management
1 48 (29.27%) Hadoop YARN
2 11 (6.71%) Oozie
3 9 (5.49%) Ansible
4 6 (3.66%) Kubernetes
5 3 (1.83%) Kibana
6 1 (0.61%) logstash
6 1 (0.61%) Opscode Chef
6 1 (0.61%) Packer
6 1 (0.61%) Prometheus
6 1 (0.61%) Puppet
6 1 (0.61%) Terraform
Vendors
1 48 (29.27%) Cloudera
2 31 (18.90%) Hortonworks
3 16 (9.76%) Informatica
3 16 (9.76%) Talend
4 15 (9.15%) Google
5 14 (8.54%) Oracle
6 7 (4.27%) MapR
7 4 (2.44%) Ab Initio
8 3 (1.83%) Databricks
9 2 (1.22%) Tableau
10 1 (0.61%) SaltStack
10 1 (0.61%) SAP