Period
to 20 July 2018

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Spark Streaming skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark Streaming over the 6 months to 20 July 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark Streaming
England > London
6 months to
20 Jul 2018
Same period 2017 Same period 2016
Rank 573 550 595
Rank change year-on-year -23 +45 +3
Contract jobs citing Spark Streaming 69 43 24
As % of all contract IT jobs advertised in London 0.15% 0.10% 0.053%
As % of the Libraries, Frameworks & Software Standards category 0.39% 0.28% 0.16%
Number of daily rates quoted 50 35 13
London median daily rate £575 £550 £575
Median daily rate % change year-on-year +4.55% -4.35% -4.17%
10th Percentile £425 £406 £498
90th Percentile £653 £663 £845
England median daily rate £525 £515 £575
% change year-on-year +1.94% -10.43% -4.17%
Number of hourly rates quoted 0 0 0
London median hourly rate - - -
England median hourly rate - £35.00 -

Spark Streaming is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
London
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 17,571 15,090 14,889
As % of all contract IT jobs advertised in London 37.79% 36.01% 32.71%
Number of daily rates quoted 13,383 11,211 11,249
London median daily rate £513 £492 £465
Median daily rate % change year-on-year +4.17% +5.81% +3.33%
10th Percentile £363 £338 £313
90th Percentile £675 £650 £638
England median daily rate £475 £450 £425
% change year-on-year +5.56% +5.88% +6.25%
Number of hourly rates quoted 120 98 52
London median hourly rate £43.25 £38.75 £29.00
Median hourly rate % change year-on-year +11.61% +33.62% +0.59%
10th Percentile £18.55 £18.40 £19.00
90th Percentile £70.81 £54.38 £57.25
England median hourly rate £41.52 £40.50 £37.50
% change year-on-year +2.51% +8.00% +4.17%

Spark Streaming
Job Vacancy Trend in London

Job postings citing Spark Streaming as a percentage of all IT jobs advertised in London.

Job vacancy trend for Spark Streaming in London

Spark Streaming
Contractor Daily Rate Trend in London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark Streaming in London.

Contractor daily rate trend for Spark Streaming in London

Spark Streaming
Contractor Daily Rate Histogram in London

The daily rate distribution of IT jobs citing Spark Streaming in London over the 6 months to 20 July 2018.

Contractor daily rate histogram for Spark Streaming in London

Spark Streaming
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark Streaming within the London region over the 6 months to 20 July 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
West London -33 8 £613 +28.95%
Central London -29 6 £650 - 2
East London -17 3 £550 -4.35%
Spark Streaming
England

For the 6 months to 20 July 2018, IT contractor jobs citing Spark Streaming also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Spark Streaming.

1 69 (100.00%) Apache Spark
2 59 (85.51%) Scala
3 58 (84.06%) Kafka
4 50 (72.46%) Big Data
5 42 (60.87%) Agile Software Development
6 38 (55.07%) Java
6 38 (55.07%) Hadoop
7 32 (46.38%) Apache Hive
7 32 (46.38%) NoSQL
8 29 (42.03%) Apache Cassandra
8 29 (42.03%) Python
9 26 (37.68%) Apache
10 22 (31.88%) Cloudera
11 21 (30.43%) SQL
12 20 (28.99%) HBase
13 19 (27.54%) Business Intelligence
13 19 (27.54%) Docker
13 19 (27.54%) Microservices
14 18 (26.09%) Machine Learning
15 17 (24.64%) RESTful
15 17 (24.64%) HDFS
15 17 (24.64%) DevOps
16 16 (23.19%) MongoDB
16 16 (23.19%) Data Modelling
16 16 (23.19%) Data Warehouse
16 16 (23.19%) TDD
17 15 (21.74%) Pair Programming
17 15 (21.74%) HTTP
17 15 (21.74%) Amazon AWS
17 15 (21.74%) Impala

Spark Streaming
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 69 (100.00%) Apache Spark
2 26 (37.68%) Apache
3 10 (14.49%) Apache Pig
3 10 (14.49%) Elasticsearch
4 9 (13.04%) Apache Airflow
5 5 (7.25%) Apache Solr
6 1 (1.45%) Luigi
Cloud Services
1 15 (21.74%) Amazon AWS
2 14 (20.29%) Google Cloud Platform
3 9 (13.04%) Amazon EMR
4 6 (8.70%) Microsoft Azure
5 5 (7.25%) GitHub
6 1 (1.45%) Amazon EC2
6 1 (1.45%) Amazon Kinesis
6 1 (1.45%) Amazon S3
Communications & Networking
1 15 (21.74%) HTTP
2 2 (2.90%) Kerberos
Database & Business Intelligence
1 50 (72.46%) Big Data
2 38 (55.07%) Hadoop
3 32 (46.38%) Apache Hive
3 32 (46.38%) NoSQL
4 29 (42.03%) Apache Cassandra
5 20 (28.99%) HBase
6 16 (23.19%) Data Warehouse
6 16 (23.19%) MongoDB
7 15 (21.74%) Impala
8 14 (20.29%) Relational Database
9 13 (18.84%) SQL Server
10 10 (14.49%) Cloudera CDH
10 10 (14.49%) Spark SQL
11 9 (13.04%) Sqoop
12 8 (11.59%) Oracle Database
12 8 (11.59%) Oracle Database 11g
12 8 (11.59%) Oracle Database 12c
13 7 (10.14%) Data Lake
14 6 (8.70%) MySQL
15 5 (7.25%) OLTP
Development Applications
1 11 (15.94%) JIRA
2 8 (11.59%) Oracle APEX
2 8 (11.59%) Team Foundation Server
3 6 (8.70%) Git (software)
4 4 (5.80%) Gatling
4 4 (5.80%) TeamCity
4 4 (5.80%) Visual Studio Team System
5 3 (4.35%) Jenkins
6 2 (2.90%) GitLab
6 2 (2.90%) Travis CI
7 1 (1.45%) Appium
7 1 (1.45%) Cucumber
7 1 (1.45%) Selenium
General
1 9 (13.04%) Finance
2 8 (11.59%) Telecoms
3 6 (8.70%) Banking
4 4 (5.80%) Games
4 4 (5.80%) Investment Banking
5 2 (2.90%) Retail Banking
6 1 (1.45%) Publishing
Job Titles
1 49 (71.01%) Developer
2 30 (43.48%) Big Data Developer
3 15 (21.74%) Scala Developer
4 12 (17.39%) Data Engineer
5 11 (15.94%) Big Data Engineer
6 10 (14.49%) Apache Spark Developer
7 8 (11.59%) Architect
7 8 (11.59%) Business Developer
7 8 (11.59%) Business Intelligence Developer
7 8 (11.59%) Data Architect
7 8 (11.59%) Python Developer
8 7 (10.14%) Big Data Architect
8 7 (10.14%) Big Data Consultant
8 7 (10.14%) Consultant
8 7 (10.14%) Data Consultant
9 6 (8.70%) Hadoop Developer
10 4 (5.80%) DevOps Engineer
10 4 (5.80%) Java Developer
10 4 (5.80%) Software Developer
10 4 (5.80%) Software Engineer
Libraries, Frameworks & Software Standards
1 58 (84.06%) Kafka
2 17 (24.64%) RESTful
3 15 (21.74%) Akka
3 15 (21.74%) Play Framework
3 15 (21.74%) spray
4 5 (7.25%) Apache Avro
4 5 (7.25%) RabbitMQ
5 3 (4.35%) TensorFlow
6 2 (2.90%) Natural Language Processing
6 2 (2.90%) OpenCV
7 1 (1.45%) Apache Thrift
7 1 (1.45%) Elastic Stack
7 1 (1.45%) Esper
7 1 (1.45%) MLlib
7 1 (1.45%) Node.js
7 1 (1.45%) Protocol Buffers
7 1 (1.45%) REST
7 1 (1.45%) Web Services
7 1 (1.45%) XML
7 1 (1.45%) XSD
Miscellaneous
1 5 (7.25%) Analytical Skills
2 4 (5.80%) Self-Motivation
3 3 (4.35%) Algorithms
4 2 (2.90%) Data Structures
4 2 (2.90%) Hedge funds
5 1 (1.45%) Data Centre
5 1 (1.45%) Greenfield Project
5 1 (1.45%) Taxonomies
Operating Systems
1 14 (20.29%) Linux
2 12 (17.39%) Unix
Processes & Methodologies
1 42 (60.87%) Agile Software Development
2 19 (27.54%) Business Intelligence
2 19 (27.54%) Microservices
3 18 (26.09%) Machine Learning
4 17 (24.64%) DevOps
5 16 (23.19%) Data Modelling
5 16 (23.19%) TDD
6 15 (21.74%) Coaching
6 15 (21.74%) Pair Programming
7 11 (15.94%) Analytics
8 8 (11.59%) ETL
9 6 (8.70%) Deep Learning
9 6 (8.70%) Scrum
9 6 (8.70%) Software Engineering
10 5 (7.25%) Data Integration
10 5 (7.25%) Data Science
10 5 (7.25%) Dimensional Modelling
11 4 (5.80%) Convolutional Neural Network
11 4 (5.80%) OOP
11 4 (5.80%) Web Development
Programming Languages
1 59 (85.51%) Scala
2 38 (55.07%) Java
3 29 (42.03%) Python
4 21 (30.43%) SQL
5 8 (11.59%) Shell Script
6 5 (7.25%) C++
7 4 (5.80%) C#
8 3 (4.35%) Bash Shell
9 1 (1.45%) Groovy
9 1 (1.45%) Java 8
Quality Assurance & Compliance
1 10 (14.49%) GCP
2 1 (1.45%) Data Quality
2 1 (1.45%) GDPR
2 1 (1.45%) PSD2
System Software
1 19 (27.54%) Docker
2 17 (24.64%) HDFS
3 7 (10.14%) Apache ZooKeeper
4 5 (7.25%) Apache Flume
Systems Management
1 13 (18.84%) Hadoop YARN
2 10 (14.49%) Oozie
3 7 (10.14%) Kubernetes
4 2 (2.90%) Ansible
4 2 (2.90%) Prometheus
5 1 (1.45%) Kibana
5 1 (1.45%) logstash
5 1 (1.45%) Opscode Chef
5 1 (1.45%) Packer
5 1 (1.45%) Puppet
5 1 (1.45%) Terraform
Vendors
1 22 (31.88%) Cloudera
2 14 (20.29%) Google
3 13 (18.84%) Oracle
4 9 (13.04%) Hortonworks
5 7 (10.14%) MapR
6 4 (5.80%) Databricks
7 1 (1.45%) SaltStack
7 1 (1.45%) SAP
7 1 (1.45%) Tableau