Period
to 18 September 2018

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Spark Streaming skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark Streaming over the 6 months to 18 September 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark Streaming
England > London
6 months to
18 Sep 2018
Same period 2017 Same period 2016
Rank 577 556 581
Rank change year-on-year -21 +25 +34
Contract jobs citing Spark Streaming 60 51 47
As % of all contract IT jobs advertised in London 0.13% 0.12% 0.11%
As % of the Libraries, Frameworks & Software Standards category 0.33% 0.31% 0.31%
Number of daily rates quoted 35 42 32
London median daily rate £625 £525 £575
Median daily rate % change year-on-year +19.05% -8.70% -
10th Percentile £435 £438 £451
90th Percentile £798 £625 £818
England median daily rate £550 £525 £550
% change year-on-year +4.76% -4.55% -
Number of hourly rates quoted 0 0 0
London median hourly rate - - -
England median hourly rate - £35.00 -

Spark Streaming is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
London
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 18,094 16,229 14,984
As % of all contract IT jobs advertised in London 39.40% 36.96% 33.79%
Number of daily rates quoted 13,670 12,162 11,295
London median daily rate £520 £500 £450
Median daily rate % change year-on-year +4.00% +11.11% -
10th Percentile £369 £350 £313
90th Percentile £675 £650 £625
England median daily rate £475 £450 £425
% change year-on-year +5.56% +5.88% +6.25%
Number of hourly rates quoted 91 105 62
London median hourly rate £43.50 £35.00 £40.00
Median hourly rate % change year-on-year +24.29% -12.50% +23.08%
10th Percentile £20.95 £18.20 £19.00
90th Percentile £72.50 £51.20 £78.43
England median hourly rate £41.81 £40.00 £36.50
% change year-on-year +4.51% +9.59% -1.35%

Spark Streaming
Job Vacancy Trend in London

Job postings citing Spark Streaming as a percentage of all IT jobs advertised in London.

Job vacancy trend for Spark Streaming in London

Spark Streaming
Contractor Daily Rate Trend in London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark Streaming in London.

Contractor daily rate trend for Spark Streaming in London

Spark Streaming
Contractor Daily Rate Histogram in London

The daily rate distribution of IT jobs citing Spark Streaming in London over the 6 months to 18 September 2018.

Contractor daily rate histogram for Spark Streaming in London

Spark Streaming
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark Streaming within the London region over the 6 months to 18 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
West London -18 9 £600 +14.29%
Central London -16 8 £650 - 5
East London -7 2 - -
Spark Streaming
England

For the 6 months to 18 September 2018, IT contractor jobs citing Spark Streaming also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Spark Streaming.

1 60 (100.00%) Apache Spark
2 54 (90.00%) Kafka
3 49 (81.67%) Big Data
3 49 (81.67%) Scala
4 42 (70.00%) Hadoop
4 42 (70.00%) Java
5 41 (68.33%) Agile Software Development
6 39 (65.00%) Apache Hive
7 37 (61.67%) HBase
7 37 (61.67%) Python
8 28 (46.67%) Apache
9 27 (45.00%) NoSQL
9 27 (45.00%) Hadoop YARN
10 26 (43.33%) Apache Cassandra
11 25 (41.67%) Data Modelling
11 25 (41.67%) Elasticsearch
12 22 (36.67%) SQL
13 20 (33.33%) Unix
13 20 (33.33%) Cloudera
13 20 (33.33%) RESTful
13 20 (33.33%) DevOps
14 19 (31.67%) ETL
15 18 (30.00%) Linux
16 17 (28.33%) Google
16 17 (28.33%) JIRA
16 17 (28.33%) Google Cloud Platform
17 16 (26.67%) Docker
17 16 (26.67%) Git (software)
17 16 (26.67%) GitHub
18 15 (25.00%) MapReduce

Spark Streaming
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 60 (100.00%) Apache Spark
2 28 (46.67%) Apache
3 25 (41.67%) Elasticsearch
4 12 (20.00%) Apache Pig
5 10 (16.67%) Apache Solr
6 4 (6.67%) Apache Airflow
7 1 (1.67%) Luigi
Cloud Services
1 17 (28.33%) Google Cloud Platform
2 16 (26.67%) GitHub
3 10 (16.67%) Amazon AWS
4 6 (10.00%) Microsoft Azure
5 5 (8.33%) Amazon EMR
6 1 (1.67%) Amazon EC2
6 1 (1.67%) Amazon Kinesis
Communications & Networking
1 9 (15.00%) HTTP
2 3 (5.00%) DNS
3 1 (1.67%) Kerberos
Database & Business Intelligence
1 49 (81.67%) Big Data
2 42 (70.00%) Hadoop
3 39 (65.00%) Apache Hive
4 37 (61.67%) HBase
5 27 (45.00%) NoSQL
6 26 (43.33%) Apache Cassandra
7 15 (25.00%) Presto
7 15 (25.00%) Relational Database
8 14 (23.33%) Impala
9 13 (21.67%) MongoDB
10 11 (18.33%) Data Warehouse
11 10 (16.67%) Spark SQL
12 8 (13.33%) Sqoop
13 7 (11.67%) Data Lake
13 7 (11.67%) SQL Server
14 5 (8.33%) Enterprise Data Warehouse
14 5 (8.33%) Flink
14 5 (8.33%) MySQL
14 5 (8.33%) OLTP
15 4 (6.67%) Parquet
Development Applications
1 17 (28.33%) JIRA
2 16 (26.67%) Git (software)
3 13 (21.67%) Jenkins
4 6 (10.00%) Gatling
5 4 (6.67%) TeamCity
5 4 (6.67%) Visual Studio Team System
6 3 (5.00%) Oracle APEX
6 3 (5.00%) Travis CI
7 2 (3.33%) GitLab
7 2 (3.33%) JUnit
7 2 (3.33%) Team Foundation Server
8 1 (1.67%) Atlassian Bamboo
8 1 (1.67%) EasyMock
8 1 (1.67%) Mockito
8 1 (1.67%) TestNG
General
1 12 (20.00%) Publishing
2 9 (15.00%) Finance
3 8 (13.33%) Banking
4 5 (8.33%) Games
5 4 (6.67%) Telecoms
6 3 (5.00%) Investment Banking
6 3 (5.00%) Retail Banking
Job Titles
1 22 (36.67%) Data Engineer
1 22 (36.67%) Developer
2 19 (31.67%) Big Data Engineer
3 13 (21.67%) Big Data Developer
4 11 (18.33%) Architect
5 10 (16.67%) Big Data Architect
5 10 (16.67%) Data Architect
6 8 (13.33%) Scala Developer
7 5 (8.33%) Hadoop Developer
8 4 (6.67%) Apache Spark Developer
8 4 (6.67%) Big Data Consultant
8 4 (6.67%) Cloud Engineer
8 4 (6.67%) Consultant
8 4 (6.67%) Data Consultant
9 3 (5.00%) Business Developer
9 3 (5.00%) Business Intelligence Developer
9 3 (5.00%) DevOps Engineer
9 3 (5.00%) Python Developer
10 2 (3.33%) Security Architect
10 2 (3.33%) Site Engineer
Libraries, Frameworks & Software Standards
1 54 (90.00%) Kafka
2 20 (33.33%) RESTful
3 10 (16.67%) Elastic Stack
4 8 (13.33%) Akka
4 8 (13.33%) Play Framework
4 8 (13.33%) spray
5 5 (8.33%) RabbitMQ
6 4 (6.67%) Apache Avro
7 3 (5.00%) TensorFlow
8 2 (3.33%) Java Concurrency
8 2 (3.33%) Lambda Expressions
8 2 (3.33%) OpenCV
8 2 (3.33%) Spring
9 1 (1.67%) PySpark
9 1 (1.67%) WebSockets
9 1 (1.67%) WireMock
Miscellaneous
1 6 (10.00%) Greenfield Project
2 4 (6.67%) Algorithms
2 4 (6.67%) Analytical Skills
2 4 (6.67%) Cyber Defence
2 4 (6.67%) Self-Motivation
3 1 (1.67%) Computer Science
3 1 (1.67%) Cyberattack
3 1 (1.67%) Data Centre
3 1 (1.67%) Data Structures
3 1 (1.67%) Hedge funds
Operating Systems
1 20 (33.33%) Unix
2 18 (30.00%) Linux
Processes & Methodologies
1 41 (68.33%) Agile Software Development
2 25 (41.67%) Data Modelling
3 20 (33.33%) DevOps
4 19 (31.67%) ETL
5 15 (25.00%) Business Intelligence
5 15 (25.00%) Deployment Automation
5 15 (25.00%) MapReduce
6 14 (23.33%) Analytics
6 14 (23.33%) Scrum
7 13 (21.67%) SDLC
7 13 (21.67%) TDD
8 12 (20.00%) Requirements Management
9 11 (18.33%) Microservices
10 8 (13.33%) Coaching
10 8 (13.33%) Data Science
10 8 (13.33%) Machine Learning
10 8 (13.33%) Pair Programming
10 8 (13.33%) Project Planning
10 8 (13.33%) Waterfall
11 7 (11.67%) Kanban
Programming Languages
1 49 (81.67%) Scala
2 42 (70.00%) Java
3 37 (61.67%) Python
4 22 (36.67%) SQL
5 5 (8.33%) Bash Shell
5 5 (8.33%) C++
6 4 (6.67%) C#
7 3 (5.00%) Shell Script
8 1 (1.67%) PL/SQL
Qualifications
1 1 (1.67%) ISTQB
1 1 (1.67%) ISTQB Advanced Certification
Quality Assurance & Compliance
1 5 (8.33%) GCP
2 1 (1.67%) QA
System Software
1 16 (26.67%) Docker
2 15 (25.00%) HDFS
3 8 (13.33%) Apache Flume
4 5 (8.33%) Apache ZooKeeper
Systems Management
1 27 (45.00%) Hadoop YARN
2 15 (25.00%) Ansible
3 10 (16.67%) Kibana
3 10 (16.67%) logstash
4 8 (13.33%) Kubernetes
5 5 (8.33%) Oozie
6 3 (5.00%) Opscode Chef
6 3 (5.00%) Puppet
7 2 (3.33%) Prometheus
8 1 (1.67%) Packer
Vendors
1 20 (33.33%) Cloudera
2 17 (28.33%) Google
3 8 (13.33%) Oracle
4 7 (11.67%) Hortonworks
4 7 (11.67%) MapR
5 4 (6.67%) Databricks
6 3 (5.00%) SaltStack