Period
to 20 September 2018

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 20 September 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
20 Sep 2018
Same period 2017 Same period 2016
Rank 79 103 133
Rank change year-on-year +24 +30 +80
Contract jobs citing Apache Spark 216 161 114
As % of all contract IT jobs advertised in the City of London 3.09% 2.37% 1.67%
As % of the Application Platforms category 17.96% 15.99% 9.60%
Number of daily rates quoted 190 144 93
City of London median daily rate £550 £600 £593
Median daily rate % change year-on-year -8.33% +1.27% -1.25%
10th Percentile £469 £472 £415
90th Percentile £726 £725 £713
Central London median daily rate £550 £600 £588
% change year-on-year -8.33% +2.13% -2.08%
Number of hourly rates quoted 0 0 0
City of London median hourly rate - - -
Central London median hourly rate - - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,203 1,007 1,188
As % of all contract IT jobs advertised in the City of London 17.21% 14.84% 17.39%
Number of daily rates quoted 907 755 923
City of London median daily rate £510 £498 £450
Median daily rate % change year-on-year +2.51% +10.56% +5.88%
10th Percentile £284 £268 £289
90th Percentile £675 £675 £638
Central London median daily rate £500 £494 £450
% change year-on-year +1.27% +9.72% +5.88%
Number of hourly rates quoted 36 8 16
City of London median hourly rate £20.63 £17.00 £18.25
Median hourly rate % change year-on-year +21.32% -6.85% +7.35%
10th Percentile £13.00 £12.01 £11.25
90th Percentile £33.38 £59.63 £41.13
Central London median hourly rate £20.00 £17.85 £17.50
% change year-on-year +12.04% +2.00% +2.94%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 20 September 2018.

Contractor daily rate histogram for Apache Spark in the City of London

Apache Spark
Contract Job Locations in the City of London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the City of London region over the 6 months to 20 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London Bridge - 37 £550 -
Apache Spark
Central London

For the 6 months to 20 September 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 123 (56.94%) Hadoop
2 116 (53.70%) Big Data
3 113 (52.31%) Python
4 102 (47.22%) Kafka
5 89 (41.20%) Java
6 76 (35.19%) Finance
7 75 (34.72%) Amazon AWS
8 70 (32.41%) Agile Software Development
9 64 (29.63%) Apache Hive
10 63 (29.17%) Scala
11 58 (26.85%) SQL
12 56 (25.93%) Data Science
13 55 (25.46%) Elasticsearch
13 55 (25.46%) Docker
14 51 (23.61%) Linux
15 47 (21.76%) Machine Learning
16 41 (18.98%) Microservices
17 39 (18.06%) NoSQL
18 38 (17.59%) MongoDB
19 37 (17.13%) Git (software)
19 37 (17.13%) Pandas
20 36 (16.67%) Apache Cassandra
21 34 (15.74%) NumPy
21 34 (15.74%) PySpark
22 32 (14.81%) Analytics
22 32 (14.81%) Apache
23 31 (14.35%) HBase
24 29 (13.43%) TDD
24 29 (13.43%) Kubernetes
24 29 (13.43%) HDFS

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 55 (25.46%) Elasticsearch
2 32 (14.81%) Apache
3 14 (6.48%) Apache Pig
4 12 (5.56%) Apache Airflow
4 12 (5.56%) Apache Solr
5 5 (2.31%) SAS
6 3 (1.39%) Luigi
6 3 (1.39%) Oracle Coherence
6 3 (1.39%) Oracle SOA Suite
7 2 (0.93%) Confluence
8 1 (0.46%) Gunicorn
8 1 (0.46%) nginx
8 1 (0.46%) SharePoint
8 1 (0.46%) Tomcat
8 1 (0.46%) WordPress
Applications
1 1 (0.46%) Microsoft Excel
1 1 (0.46%) Microsoft Office
1 1 (0.46%) Microsoft PowerPoint
1 1 (0.46%) MS Visio
Cloud Services
1 75 (34.72%) Amazon AWS
2 24 (11.11%) Microsoft Azure
3 22 (10.19%) Heroku
4 18 (8.33%) Google Cloud Platform
5 8 (3.70%) Amazon EMR
6 6 (2.78%) GitHub
7 5 (2.31%) Amazon S3
7 5 (2.31%) AWS Lambda
7 5 (2.31%) Serverless
8 3 (1.39%) Amazon Kinesis
8 3 (1.39%) IBM Cloud
9 2 (0.93%) Azure Data Factory
9 2 (0.93%) Cloud Computing
10 1 (0.46%) Azure Machine Learning
10 1 (0.46%) Data Lake Analytics
10 1 (0.46%) Route 53
Communications & Networking
1 28 (12.96%) SSL
2 24 (11.11%) HTTP
3 22 (10.19%) DNS
3 22 (10.19%) FTP
3 22 (10.19%) TCP/IP
4 8 (3.70%) Kerberos
5 1 (0.46%) DSL
5 1 (0.46%) Internet
5 1 (0.46%) SNA
Database & Business Intelligence
1 123 (56.94%) Hadoop
2 116 (53.70%) Big Data
3 64 (29.63%) Apache Hive
4 39 (18.06%) NoSQL
5 38 (17.59%) MongoDB
6 36 (16.67%) Apache Cassandra
7 31 (14.35%) HBase
8 27 (12.50%) PostgreSQL
9 22 (10.19%) Data Warehouse
10 17 (7.87%) Amazon Redshift
11 16 (7.41%) Relational Database
12 13 (6.02%) Redis
12 13 (6.02%) SQL Server
13 11 (5.09%) Sqoop
14 10 (4.63%) OLAP
14 10 (4.63%) Star Schema
15 9 (4.17%) Enterprise Data Warehouse
15 9 (4.17%) OLAP Cube
15 9 (4.17%) SQL Server Integration Services
16 8 (3.70%) Data Lake
Development Applications
1 37 (17.13%) Git (software)
2 28 (12.96%) GitLab
3 16 (7.41%) Jenkins
4 12 (5.56%) Gradle
4 12 (5.56%) gulp
5 5 (2.31%) JIRA
5 5 (2.31%) Subversion
5 5 (2.31%) Travis CI
6 4 (1.85%) Maven
7 3 (1.39%) Atlassian Bamboo
7 3 (1.39%) JUnit
7 3 (1.39%) Visual Studio Team System
8 2 (0.93%) FishEye
8 2 (0.93%) GoCD
8 2 (0.93%) jMock
8 2 (0.93%) Visual Studio
9 1 (0.46%) Jasmine
9 1 (0.46%) Karma Test Runner
9 1 (0.46%) SQL Server Management Studio (SSMS)
9 1 (0.46%) webpack
General
1 76 (35.19%) Finance
2 12 (5.56%) Banking
3 6 (2.78%) Advertising
3 6 (2.78%) Games
3 6 (2.78%) Retail
4 5 (2.31%) Front Office
5 3 (1.39%) Investment Banking
5 3 (1.39%) Marketing
6 2 (0.93%) Prime Brokerage
6 2 (0.93%) Publishing
7 1 (0.46%) Electronics
7 1 (0.46%) Financial Institution
7 1 (0.46%) Law
7 1 (0.46%) Telecoms
Job Titles
1 102 (47.22%) Developer
2 62 (28.70%) Data Engineer
3 36 (16.67%) Python Developer
4 34 (15.74%) Apache Spark Developer
5 32 (14.81%) Big Data Engineer
6 25 (11.57%) Java Developer
7 22 (10.19%) Elasticsearch Developer
7 22 (10.19%) Senior Developer
8 21 (9.72%) Big Data Developer
9 17 (7.87%) Python Engineer
10 14 (6.48%) Hadoop Engineer
11 12 (5.56%) Senior Java Developer
12 11 (5.09%) AWS Engineer
12 11 (5.09%) Scala Developer
13 10 (4.63%) Ruby Developer
13 10 (4.63%) Senior Data Engineer
14 9 (4.17%) Consultant
15 8 (3.70%) Lead Developer
15 8 (3.70%) Lead Java Developer
15 8 (3.70%) Technical Leader
Libraries, Frameworks & Software Standards
1 102 (47.22%) Kafka
2 37 (17.13%) Pandas
3 34 (15.74%) NumPy
3 34 (15.74%) PySpark
4 27 (12.50%) REST
5 25 (11.57%) AngularJS
6 24 (11.11%) HTML
7 23 (10.65%) CSS
7 23 (10.65%) HTML5
8 22 (10.19%) OAuth
8 22 (10.19%) OAuth2
8 22 (10.19%) SOAP
9 17 (7.87%) RabbitMQ
10 16 (7.41%) Spring
11 13 (6.02%) React
12 12 (5.56%) Dropwizard
12 12 (5.56%) Ember.js
12 12 (5.56%) Guice
12 12 (5.56%) Ratpack
12 12 (5.56%) Spring Boot
Miscellaneous
1 26 (12.04%) Foreign Exchange (FX)
2 22 (10.19%) Algorithms
3 20 (9.26%) Fintech
4 15 (6.94%) Greenfield Project
5 10 (4.63%) Credit Risk
6 6 (2.78%) Computer Science
7 5 (2.31%) Analytical Skills
8 4 (1.85%) Market Risk
9 3 (1.39%) Data Structures
9 3 (1.39%) Derivative
9 3 (1.39%) Fixed Income
10 2 (0.93%) Cloud Native
10 2 (0.93%) Enterprise Software
10 2 (0.93%) Linux Command Line
10 2 (0.93%) Mobile App
10 2 (0.93%) Online Games
10 2 (0.93%) OTC Derivatives
10 2 (0.93%) Private Cloud
10 2 (0.93%) Virtual Team
11 1 (0.46%) Social Network
Operating Systems
1 51 (23.61%) Linux
2 9 (4.17%) Unix
3 8 (3.70%) Red Hat Enterprise Linux
4 2 (0.93%) Apple iOS
4 2 (0.93%) DC/OS
5 1 (0.46%) Mac OS X
5 1 (0.46%) Ubuntu
Processes & Methodologies
1 70 (32.41%) Agile Software Development
2 56 (25.93%) Data Science
3 47 (21.76%) Machine Learning
4 41 (18.98%) Microservices
5 32 (14.81%) Analytics
6 29 (13.43%) TDD
7 28 (12.96%) ETL
7 28 (12.96%) Software Engineering
8 25 (11.57%) Data Modelling
9 22 (10.19%) Open Source
9 22 (10.19%) OWASP
10 21 (9.72%) BDD
10 21 (9.72%) DevOps
11 18 (8.33%) Scrum
12 17 (7.87%) Data Engineering
13 16 (7.41%) Continuous Integration
14 15 (6.94%) Visualisation
15 14 (6.48%) MapReduce
16 13 (6.02%) Business Intelligence
16 13 (6.02%) Data Visualisation
Programming Languages
1 113 (52.31%) Python
2 89 (41.20%) Java
3 63 (29.17%) Scala
4 58 (26.85%) SQL
5 27 (12.50%) Java 8
6 25 (11.57%) JavaScript
7 22 (10.19%) ES6
8 18 (8.33%) R
9 14 (6.48%) TypeScript
10 11 (5.09%) Ruby
11 8 (3.70%) Bash Shell
12 6 (2.78%) Clojure
12 6 (2.78%) MATLAB
13 5 (2.31%) T-SQL
14 3 (1.39%) C
14 3 (1.39%) Kotlin
14 3 (1.39%) PL/SQL
15 2 (0.93%) C#
15 2 (0.93%) Cypher
15 2 (0.93%) Julia
Qualifications
1 8 (3.70%) Degree
2 3 (1.39%) Computer Science Degree
3 2 (0.93%) Numerate Degree
4 1 (0.46%) ISTQB
4 1 (0.46%) ISTQB Advanced Certification
4 1 (0.46%) Master's Degree
4 1 (0.46%) Microsoft Certification
4 1 (0.46%) PhD
4 1 (0.46%) SC Cleared
4 1 (0.46%) Security Cleared
Quality Assurance & Compliance
1 10 (4.63%) Data Quality
2 7 (3.24%) GCP
2 7 (3.24%) QA
3 1 (0.46%) GDPR
System Software
1 55 (25.46%) Docker
2 29 (13.43%) HDFS
3 16 (7.41%) Apache Flume
4 9 (4.17%) Apache ZooKeeper
5 6 (2.78%) Active Directory
Systems Management
1 29 (13.43%) Kubernetes
2 24 (11.11%) Oozie
3 19 (8.80%) Hadoop YARN
4 12 (5.56%) Docker Swarm
4 12 (5.56%) Mesos
5 10 (4.63%) Marathon
6 9 (4.17%) Ambari
6 9 (4.17%) Ansible
7 6 (2.78%) Apache Slider
8 5 (2.31%) Puppet
8 5 (2.31%) Terraform
9 3 (1.39%) Kibana
10 2 (0.93%) logstash
10 2 (0.93%) Opscode Chef
10 2 (0.93%) Prometheus
10 2 (0.93%) rkt
Vendors
1 28 (12.96%) Splunk
2 19 (8.80%) Oracle
3 17 (7.87%) Google
4 11 (5.09%) Hortonworks
4 11 (5.09%) Tableau
5 10 (4.63%) Cloudera
6 8 (3.70%) Informatica
7 7 (3.24%) Talend
8 6 (2.78%) Microsoft
9 5 (2.31%) IBM
10 4 (1.85%) Alteryx
10 4 (1.85%) Sybase
11 3 (1.39%) Ab Initio
11 3 (1.39%) Red Hat
11 3 (1.39%) TIBCO
12 2 (0.93%) Atlassian
12 2 (0.93%) MapR
13 1 (0.46%) Palantir
13 1 (0.46%) Teradata