Period
to 18 September 2018

The following table provides summary statistics for contract job vacancies with a requirement for Spark Streaming skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Spark Streaming over the 6 months to 18 September 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark Streaming
UK
6 months to
18 Sep 2018
Same period 2017 Same period 2016
Rank 741 746 828
Rank change year-on-year +5 +82 +67
Contract jobs citing Spark Streaming 142 112 64
As % of all contract IT jobs advertised in the UK 0.14% 0.11% 0.060%
As % of the Libraries, Frameworks & Software Standards category 0.39% 0.35% 0.20%
Number of daily rates quoted 92 73 37
UK median daily rate £544 £525 £550
Median daily rate % change year-on-year +3.57% -4.55% -
10th Percentile £450 £419 £455
90th Percentile £675 £585 £780
UK excluding London median daily rate £525 £465 £550
% change year-on-year +12.90% -15.45% -
Number of hourly rates quoted 0 1 0
UK median hourly rate - £35.00 -
UK excluding London median hourly rate - £35.00 -

Spark Streaming is in the Software Libraries and Frameworks category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for technical specification, industry standards, software libraries and framework skills.

All Software Libraries and Frameworks
UK
Contract vacancies with a requirement for technical specification, industry standards, software libraries and framework skills 36,058 31,745 31,662
As % of all contract IT jobs advertised in the UK 34.99% 32.41% 29.79%
Number of daily rates quoted 25,414 21,983 22,160
UK median daily rate £475 £450 £425
Median daily rate % change year-on-year +5.56% +5.88% +6.25%
10th Percentile £325 £310 £294
90th Percentile £625 £625 £588
UK excluding London median daily rate £425 £400 £388
% change year-on-year +6.25% +3.23% +3.33%
Number of hourly rates quoted 339 394 382
UK median hourly rate £41.53 £40.00 £36.00
Median hourly rate % change year-on-year +3.83% +11.11% -3.87%
10th Percentile £15.24 £16.00 £13.03
90th Percentile £58.00 £53.53 £50.00
UK excluding London median hourly rate £41.00 £40.00 £36.00
% change year-on-year +2.50% +11.11% -4.00%

Spark Streaming
Job Vacancy Trend

Job postings citing Spark Streaming as a percentage of all IT jobs advertised.

Job vacancy trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Trend

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Spark Streaming.

Contractor daily rate trend for Spark Streaming in the UK

Spark Streaming
Contractor Daily Rate Histogram

The daily rate distribution of IT jobs citing Spark Streaming over the 6 months to 18 September 2018.

Contractor daily rate histogram for Spark Streaming in the UK

Spark Streaming
Top 11 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Spark Streaming within the UK over the 6 months to 18 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -7 138 £550 +4.76% 13
UK excluding London -10 80 £525 +12.90% 9
South East +47 66 £525 +16.67% 4
London -21 60 £625 +19.05% 7
North of England -41 7 £525 +13.88% 1
North West - 5 £525 - 1
Scotland - 3 £431 - 2
East of England -11 2 - -
Yorkshire -48 2 £525 +13.88%
East Midlands - 2 £350 - 1
Midlands - 2 £350 - 1

For the 6 months to 18 September 2018, IT contractor jobs citing Spark Streaming also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Spark Streaming.

1 142 (100.00%) Apache Spark
2 119 (83.80%) Kafka
3 118 (83.10%) Big Data
4 103 (72.54%) Hadoop
5 100 (70.42%) Java
6 81 (57.04%) Agile Software Development
7 70 (49.30%) HBase
8 67 (47.18%) Apache Hive
8 67 (47.18%) Scala
9 64 (45.07%) Apache
9 64 (45.07%) Python
10 58 (40.85%) Hadoop YARN
11 51 (35.92%) RESTful
12 49 (34.51%) Unix
13 47 (33.10%) Finance
14 46 (32.39%) Jenkins
15 45 (31.69%) RabbitMQ
15 45 (31.69%) Linux
16 44 (30.99%) HDFS
17 41 (28.87%) TDD
18 40 (28.17%) NoSQL
19 39 (27.46%) SQL
20 38 (26.76%) Spring
20 38 (26.76%) Apache ZooKeeper
21 36 (25.35%) Apache Cassandra
21 36 (25.35%) ETL
22 35 (24.65%) Elasticsearch
23 32 (22.54%) BDD
24 30 (21.13%) Analytics
25 29 (20.42%) Data Modelling

Spark Streaming
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 142 (100.00%) Apache Spark
2 64 (45.07%) Apache
3 35 (24.65%) Elasticsearch
4 18 (12.68%) Apache Pig
5 13 (9.15%) GemFire
6 11 (7.75%) Apache Solr
7 5 (3.52%) Apache Airflow
8 2 (1.41%) Cloud Foundry
9 1 (0.70%) Luigi
Cloud Services
1 19 (13.38%) GitHub
1 19 (13.38%) Google Cloud Platform
2 18 (12.68%) Amazon AWS
3 11 (7.75%) Microsoft Azure
4 7 (4.93%) Amazon EMR
4 7 (4.93%) Amazon Kinesis
5 2 (1.41%) Amazon S3
5 2 (1.41%) PaaS
6 1 (0.70%) Amazon EC2
6 1 (0.70%) AWS Lambda
6 1 (0.70%) Serverless
Communications & Networking
1 12 (8.45%) SSH
2 11 (7.75%) Firewall
2 11 (7.75%) Internet
2 11 (7.75%) TCP/IP
3 10 (7.04%) Kerberos
4 9 (6.34%) HTTP
5 5 (3.52%) DNS
6 2 (1.41%) SSL
7 1 (0.70%) DSL
Database & Business Intelligence
1 118 (83.10%) Big Data
2 103 (72.54%) Hadoop
3 70 (49.30%) HBase
4 67 (47.18%) Apache Hive
5 40 (28.17%) NoSQL
6 36 (25.35%) Apache Cassandra
7 25 (17.61%) MongoDB
8 22 (15.49%) Relational Database
9 21 (14.79%) Impala
10 17 (11.97%) Presto
11 16 (11.27%) Flink
12 14 (9.86%) Data Warehouse
12 14 (9.86%) Sqoop
13 11 (7.75%) MySQL
13 11 (7.75%) Spark SQL
14 9 (6.34%) Data Lake
14 9 (6.34%) SQL Server
15 7 (4.93%) OLTP
16 6 (4.23%) Enterprise Data Warehouse
17 5 (3.52%) DataStax
Development Applications
1 46 (32.39%) Jenkins
2 27 (19.01%) Git (software)
3 17 (11.97%) JIRA
4 6 (4.23%) Gatling
5 4 (2.82%) JUnit
5 4 (2.82%) TeamCity
5 4 (2.82%) Visual Studio Team System
6 3 (2.11%) Oracle APEX
6 3 (2.11%) Travis CI
7 2 (1.41%) Eclipse
7 2 (1.41%) GitLab
7 2 (1.41%) Gradle
7 2 (1.41%) IntelliJ
7 2 (1.41%) Team Foundation Server
8 1 (0.70%) Atlassian Bamboo
8 1 (0.70%) EasyMock
8 1 (0.70%) Mockito
8 1 (0.70%) TestNG
General
1 47 (33.10%) Finance
2 16 (11.27%) Banking
3 12 (8.45%) Publishing
4 11 (7.75%) Legal
5 8 (5.63%) Telecoms
6 5 (3.52%) Games
6 5 (3.52%) Retail Banking
7 4 (2.82%) Investment Banking
Job Titles
1 84 (59.15%) Developer
2 45 (31.69%) Big Data Developer
3 36 (25.35%) Data Engineer
4 35 (24.65%) Java Developer
5 25 (17.61%) Big Data Engineer
6 23 (16.20%) Apache Spark Developer
7 20 (14.08%) Hadoop Developer
8 14 (9.86%) Architect
9 13 (9.15%) Big Data Architect
9 13 (9.15%) Data Architect
10 12 (8.45%) Java Hadoop Developer
11 9 (6.34%) Scala Developer
12 7 (4.93%) Analytics Developer
13 5 (3.52%) AWS Engineer
13 5 (3.52%) Software Developer
14 4 (2.82%) Big Data Consultant
14 4 (2.82%) Consultant
14 4 (2.82%) Data Consultant
14 4 (2.82%) Python Developer
14 4 (2.82%) Software Engineer
Libraries, Frameworks & Software Standards
1 119 (83.80%) Kafka
2 51 (35.92%) RESTful
3 45 (31.69%) RabbitMQ
4 38 (26.76%) Spring
5 26 (18.31%) J2EE
6 24 (16.90%) JSON
6 24 (16.90%) Middleware
7 20 (14.08%) Spring Boot
8 12 (8.45%) Elastic Stack
9 9 (6.34%) Spring Integration
10 8 (5.63%) Akka
10 8 (5.63%) Play Framework
10 8 (5.63%) spray
11 7 (4.93%) Web Services
12 6 (4.23%) ActiveMQ
13 4 (2.82%) Apache Avro
13 4 (2.82%) CSS
13 4 (2.82%) HTML
14 3 (2.11%) Node.js
14 3 (2.11%) TensorFlow
Miscellaneous
1 17 (11.97%) Analytical Skills
1 17 (11.97%) Greenfield Project
2 10 (7.04%) Computer Science
3 9 (6.34%) Algorithms
4 4 (2.82%) Cyber Defence
4 4 (2.82%) Self-Motivation
5 3 (2.11%) Data Structures
6 2 (1.41%) Fintech
6 2 (1.41%) Linux Command Line
6 2 (1.41%) Virtual Team
7 1 (0.70%) Cyberattack
7 1 (0.70%) Data Centre
7 1 (0.70%) Hedge funds
Operating Systems
1 49 (34.51%) Unix
2 45 (31.69%) Linux
3 1 (0.70%) Windows
Processes & Methodologies
1 81 (57.04%) Agile Software Development
2 41 (28.87%) TDD
3 36 (25.35%) ETL
4 32 (22.54%) BDD
5 30 (21.13%) Analytics
6 29 (20.42%) Data Modelling
7 24 (16.90%) DevOps
7 24 (16.90%) MapReduce
8 23 (16.20%) Microservices
9 20 (14.08%) Business Intelligence
9 20 (14.08%) Open Source
9 20 (14.08%) Scrum
10 18 (12.68%) Kalman Filter
11 16 (11.27%) Deployment Automation
11 16 (11.27%) Unit Testing
12 15 (10.56%) Data Ingestion
12 15 (10.56%) Web Development
13 14 (9.86%) Continuous Integration
13 14 (9.86%) Machine Learning
14 13 (9.15%) SDLC
Programming Languages
1 100 (70.42%) Java
2 67 (47.18%) Scala
3 64 (45.07%) Python
4 39 (27.46%) SQL
5 9 (6.34%) Shell Script
6 7 (4.93%) C++
7 5 (3.52%) Bash Shell
7 5 (3.52%) C#
8 4 (2.82%) Java 8
8 4 (2.82%) JavaScript
8 4 (2.82%) PHP
9 3 (2.11%) PL/SQL
10 1 (0.70%) C
10 1 (0.70%) HiveQL
10 1 (0.70%) Perl
Qualifications
1 9 (6.34%) Degree
2 2 (1.41%) Computer Science Degree
3 1 (0.70%) ISTQB
3 1 (0.70%) ISTQB Advanced Certification
3 1 (0.70%) Security Cleared
Quality Assurance & Compliance
1 5 (3.52%) GCP
2 3 (2.11%) QA
System Software
1 44 (30.99%) HDFS
2 38 (26.76%) Apache ZooKeeper
3 17 (11.97%) Docker
4 13 (9.15%) Apache Flume
Systems Management
1 58 (40.85%) Hadoop YARN
2 17 (11.97%) Ansible
3 12 (8.45%) Kibana
4 10 (7.04%) logstash
5 8 (5.63%) Kubernetes
6 6 (4.23%) Oozie
7 4 (2.82%) Opscode Chef
7 4 (2.82%) Puppet
8 2 (1.41%) Control-M
8 2 (1.41%) Prometheus
8 2 (1.41%) Tivoli
9 1 (0.70%) Packer
9 1 (0.70%) Terraform
Vendors
1 24 (16.90%) Cloudera
2 19 (13.38%) Google
3 18 (12.68%) Hortonworks
4 10 (7.04%) Oracle
5 7 (4.93%) MapR
6 4 (2.82%) Databricks
6 4 (2.82%) SaltStack
6 4 (2.82%) Tableau
7 2 (1.41%) Informatica
7 2 (1.41%) Talend