Period
to 21 February 2018

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 21 February 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
21 Feb 2018
Same period 2017 Same period 2016
Rank 105 81 154
Rank change year-on-year -24 +73 +100
Contract jobs citing Apache Spark 166 183 70
As % of all contract IT jobs advertised in the City of London 2.34% 2.97% 1.06%
As % of the Application Platforms category 15.96% 16.00% 6.99%
Number of daily rates quoted 150 161 64
City of London median daily rate £575 £600 £569
Median daily rate % change year-on-year -4.17% +5.49% +3.41%
10th Percentile £472 £438 £423
90th Percentile £725 £688 £675
Central London median daily rate £575 £600 £563
% change year-on-year -4.17% +6.67% +2.27%
Number of hourly rates quoted 0 0 1
City of London median hourly rate - - £37.50
Central London median hourly rate - - £37.50

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,040 1,144 1,001
As % of all contract IT jobs advertised in the City of London 14.65% 18.55% 15.09%
Number of daily rates quoted 839 913 746
City of London median daily rate £500 £475 £438
Median daily rate % change year-on-year +5.26% +8.57% +9.38%
10th Percentile £278 £311 £263
90th Percentile £675 £675 £650
Central London median daily rate £496 £475 £425
% change year-on-year +4.42% +11.76% +6.25%
Number of hourly rates quoted 9 11 9
City of London median hourly rate £20.00 £19.00 £19.50
Median hourly rate % change year-on-year +5.26% -2.56% +11.43%
10th Percentile £11.54 £13.00 £14.75
90th Percentile £27.50 £56.87 £60.00
Central London median hourly rate £19.35 £19.00 £19.50
% change year-on-year +1.84% -2.56% +10.28%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 21 February 2018.

Contractor daily rate histogram for Apache Spark in the City of London

For the 6 months to 21 February 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 132 (79.52%) Big Data
2 117 (70.48%) Hadoop
3 105 (63.25%) Python
4 99 (59.64%) Java
5 77 (46.39%) Kafka
6 74 (44.58%) Apache Hive
7 60 (36.14%) Scala
8 58 (34.94%) Agile Software Development
9 57 (34.34%) SQL
10 51 (30.72%) Analytics
11 49 (29.52%) HDFS
12 45 (27.11%) DevOps
13 43 (25.90%) Banking
14 42 (25.30%) Finance
15 40 (24.10%) Apache Flume
15 40 (24.10%) Data Engineering
16 39 (23.49%) Amazon AWS
17 35 (21.08%) Oozie
18 34 (20.48%) MongoDB
18 34 (20.48%) HBase
19 32 (19.28%) MapReduce
20 31 (18.67%) NoSQL
21 30 (18.07%) Apache Cassandra
22 29 (17.47%) Open Source
22 29 (17.47%) Machine Learning
22 29 (17.47%) Oracle
22 29 (17.47%) ETL
23 28 (16.87%) Data Science
24 27 (16.27%) Docker
24 27 (16.27%) TDD

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 23 (13.86%) Apache
2 11 (6.63%) Apache Pig
3 8 (4.82%) Apache Solr
3 8 (4.82%) Elasticsearch
4 6 (3.61%) Tomcat
5 5 (3.01%) nginx
5 5 (3.01%) SAS
6 4 (2.41%) SharePoint
7 3 (1.81%) Oracle Coherence
8 2 (1.20%) IPython
8 2 (1.20%) Jupyter
8 2 (1.20%) OpenStack
9 1 (0.60%) Confluence
9 1 (0.60%) IIS
9 1 (0.60%) Jetty
Applications
1 4 (2.41%) SPSS
2 2 (1.20%) Weka
Cloud Services
1 39 (23.49%) Amazon AWS
2 14 (8.43%) Microsoft Azure
3 11 (6.63%) GitHub
4 6 (3.61%) Bluemix
4 6 (3.61%) Google Cloud Platform
4 6 (3.61%) IBM Cloud
5 5 (3.01%) Amazon Kinesis
6 3 (1.81%) Cloud Computing
6 3 (1.81%) PaaS
7 2 (1.20%) Amazon S3
7 2 (1.20%) AWS Lambda
7 2 (1.20%) SaaS
7 2 (1.20%) Serverless
8 1 (0.60%) Amazon EC2
8 1 (0.60%) Amazon EMR
8 1 (0.60%) AWS CloudTrail
8 1 (0.60%) Azure Active Directory
8 1 (0.60%) Route 53
8 1 (0.60%) Virtual Private Cloud
8 1 (0.60%) Yarn
Communications & Networking
1 9 (5.42%) Kerberos
2 3 (1.81%) SSL
3 1 (0.60%) HTTP
3 1 (0.60%) Internet
Database & Business Intelligence
1 132 (79.52%) Big Data
2 117 (70.48%) Hadoop
3 74 (44.58%) Apache Hive
4 34 (20.48%) HBase
4 34 (20.48%) MongoDB
5 31 (18.67%) NoSQL
6 30 (18.07%) Apache Cassandra
7 25 (15.06%) Data Warehouse
8 21 (12.65%) PostgreSQL
8 21 (12.65%) Relational Database
9 17 (10.24%) Sqoop
10 15 (9.04%) Impala
10 15 (9.04%) RDBMS
11 13 (7.83%) Oracle Database
12 12 (7.23%) SQL Server
13 6 (3.61%) Amazon Redshift
13 6 (3.61%) Data Lake
13 6 (3.61%) Data Mining
13 6 (3.61%) Flink
13 6 (3.61%) Oracle ODI
Development Applications
1 24 (14.46%) Jenkins
2 18 (10.84%) Git (software)
3 10 (6.02%) TeamCity
4 6 (3.61%) Maven
5 5 (3.01%) JUnit
6 4 (2.41%) Cucumber
6 4 (2.41%) GoCD
6 4 (2.41%) SonarQube
7 3 (1.81%) Subversion
7 3 (1.81%) Travis CI
8 2 (1.20%) GitLab
8 2 (1.20%) Vagrant
9 1 (0.60%) Babel
9 1 (0.60%) Bower
9 1 (0.60%) Gradle
9 1 (0.60%) Grunt
9 1 (0.60%) JConsole
9 1 (0.60%) ProC
9 1 (0.60%) Visual Studio
9 1 (0.60%) VisualVM
General
1 43 (25.90%) Banking
2 42 (25.30%) Finance
3 17 (10.24%) Investment Banking
4 8 (4.82%) Retail
5 7 (4.22%) Front Office
6 5 (3.01%) Advertising
7 4 (2.41%) Publishing
8 3 (1.81%) Marketing
9 2 (1.20%) Telecoms
10 1 (0.60%) Financial Institution
10 1 (0.60%) Games
Job Titles
1 53 (31.93%) Data Engineer
2 40 (24.10%) Developer
3 31 (18.67%) Big Data Engineer
4 19 (11.45%) Big Data Developer
5 15 (9.04%) Scala Developer
6 14 (8.43%) Java Engineer
7 13 (7.83%) Data Scientist
8 11 (6.63%) DevOps Engineer
9 10 (6.02%) Java Developer
10 8 (4.82%) Apache Spark Developer
10 8 (4.82%) Hadoop Engineer
11 7 (4.22%) Architect
12 6 (3.61%) AWS Engineer
13 5 (3.01%) AWS DevOps
13 5 (3.01%) Python Developer
14 4 (2.41%) Administrator
14 4 (2.41%) AWS DevOps Engineer
14 4 (2.41%) Consultant
14 4 (2.41%) Java Programmer
14 4 (2.41%) Programmer
Libraries, Frameworks & Software Standards
1 77 (46.39%) Kafka
2 23 (13.86%) PySpark
3 17 (10.24%) Akka
4 11 (6.63%) React
5 9 (5.42%) AngularJS
5 9 (5.42%) Redux
6 8 (4.82%) Twitter Bootstrap
7 7 (4.22%) scikit-learn
7 7 (4.22%) TensorFlow
8 6 (3.61%) Keras
9 5 (3.01%) LDAP
9 5 (3.01%) MLlib
9 5 (3.01%) Natural Language Processing
9 5 (3.01%) OpenCL
9 5 (3.01%) RESTful
10 4 (2.41%) Apache Avro
10 4 (2.41%) Gherkin
10 4 (2.41%) JSON
10 4 (2.41%) REST
10 4 (2.41%) Sensu
Miscellaneous
1 18 (10.84%) Algorithms
1 18 (10.84%) Analytical Skills
2 9 (5.42%) Derivative
3 7 (4.22%) Computer Science
3 7 (4.22%) Market Risk
4 6 (3.61%) Data Structures
5 4 (2.41%) Data Centre
6 3 (1.81%) Credit Risk
6 3 (1.81%) Grid Computing
6 3 (1.81%) Interest Rate Derivative
7 2 (1.20%) Blog
7 2 (1.20%) Fintech
7 2 (1.20%) Foreign Exchange (FX)
7 2 (1.20%) Greenfield Project
7 2 (1.20%) iPad
7 2 (1.20%) Life Science
7 2 (1.20%) Replication
8 1 (0.60%) Analytical Mindset
8 1 (0.60%) Infotainment
8 1 (0.60%) Public Cloud
Operating Systems
1 25 (15.06%) Linux
2 9 (5.42%) Red Hat Enterprise Linux
3 7 (4.22%) Unix
4 2 (1.20%) DC/OS
4 2 (1.20%) Windows
4 2 (1.20%) Windows Server
4 2 (1.20%) Windows Server 2008
Processes & Methodologies
1 58 (34.94%) Agile Software Development
2 51 (30.72%) Analytics
3 45 (27.11%) DevOps
4 40 (24.10%) Data Engineering
5 32 (19.28%) MapReduce
6 29 (17.47%) ETL
6 29 (17.47%) Machine Learning
6 29 (17.47%) Open Source
7 28 (16.87%) Data Science
8 27 (16.27%) TDD
9 21 (12.65%) Business Intelligence
10 16 (9.64%) Continuous Integration
11 15 (9.04%) Data Modelling
12 14 (8.43%) Visualisation
13 13 (7.83%) BDD
13 13 (7.83%) Software Engineering
14 11 (6.63%) Data Analytics
15 10 (6.02%) Deep Learning
15 10 (6.02%) E-Commerce
15 10 (6.02%) Scrum
Programming Languages
1 105 (63.25%) Python
2 99 (59.64%) Java
3 60 (36.14%) Scala
4 57 (34.34%) SQL
5 19 (11.45%) Shell Script
6 17 (10.24%) R
7 12 (7.23%) JavaScript
8 10 (6.02%) Bash Shell
8 10 (6.02%) PL/SQL
9 7 (4.22%) C#
10 4 (2.41%) Haskell
10 4 (2.41%) PowerShell
11 3 (1.81%) MATLAB
11 3 (1.81%) Perl
12 2 (1.20%) AWK
12 2 (1.20%) Go
12 2 (1.20%) Ruby
12 2 (1.20%) sed
13 1 (0.60%) ES6
13 1 (0.60%) TypeScript
Qualifications
1 7 (4.22%) Degree
2 4 (2.41%) Computer Science Degree
3 3 (1.81%) Numerate Degree
4 2 (1.20%) Master's Degree
4 2 (1.20%) SC Cleared
4 2 (1.20%) Security Cleared
5 1 (0.60%) MCSD
5 1 (0.60%) Microsoft Certification
5 1 (0.60%) PRINCE2 Certification
Quality Assurance & Compliance
1 3 (1.81%) GCP
1 3 (1.81%) IFRS
1 3 (1.81%) IFRS 9
2 1 (0.60%) QA
System Software
1 49 (29.52%) HDFS
2 40 (24.10%) Apache Flume
3 27 (16.27%) Docker
4 11 (6.63%) Apache ZooKeeper
5 3 (1.81%) Active Directory
6 1 (0.60%) Virtual Machines
Systems Management
1 35 (21.08%) Oozie
2 19 (11.45%) Ambari
3 16 (9.64%) Hadoop YARN
4 13 (7.83%) Kubernetes
5 11 (6.63%) Ansible
6 10 (6.02%) Mesos
7 5 (3.01%) Puppet
8 4 (2.41%) Salt
8 4 (2.41%) Terraform
9 3 (1.81%) Apache Slider
10 2 (1.20%) Kibana
10 2 (1.20%) logstash
10 2 (1.20%) Rundeck
11 1 (0.60%) Opscode Chef
Vendors
1 29 (17.47%) Oracle
2 23 (13.86%) Tableau
3 18 (10.84%) Cloudera
3 18 (10.84%) Hortonworks
4 8 (4.82%) MapR
4 8 (4.82%) Microsoft
5 6 (3.61%) IBM
6 4 (2.41%) Google
6 4 (2.41%) Hitachi
6 4 (2.41%) SaltStack
6 4 (2.41%) Vertica
7 3 (1.81%) SAP
7 3 (1.81%) Talend
8 2 (1.20%) Cognos
8 2 (1.20%) Informatica
8 2 (1.20%) Salesforce.com
8 2 (1.20%) Splunk
9 1 (0.60%) Facebook
9 1 (0.60%) Netezza
9 1 (0.60%) Red Hat