Period
to 20 November 2018

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 20 November 2018 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
20 Nov 2018
Same period 2017 Same period 2016
Rank 79 98 105
Rank change year-on-year +19 +7 +78
Contract jobs citing Apache Spark 221 172 160
As % of all contract IT jobs advertised in the City of London 3.13% 2.43% 2.43%
As % of the Application Platforms category 17.53% 16.63% 13.70%
Number of daily rates quoted 190 158 135
City of London median daily rate £575 £600 £600
Median daily rate % change year-on-year -4.17% - -
10th Percentile £509 £496 £438
90th Percentile £726 £725 £688
Central London median daily rate £575 £590 £590
% change year-on-year -2.54% - -1.67%
Number of hourly rates quoted 0 0 0
City of London median hourly rate - - -
Central London median hourly rate - - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,261 1,034 1,168
As % of all contract IT jobs advertised in the City of London 17.87% 14.58% 17.71%
Number of daily rates quoted 948 809 918
City of London median daily rate £513 £500 £450
Median daily rate % change year-on-year +2.50% +11.11% +5.88%
10th Percentile £271 £278 £306
90th Percentile £663 £675 £650
Central London median daily rate £500 £498 £450
% change year-on-year +0.50% +10.56% +5.88%
Number of hourly rates quoted 33 8 15
City of London median hourly rate £21.25 £17.00 £19.00
Median hourly rate % change year-on-year +25.00% -10.53% +5.67%
10th Percentile £13.20 £10.88 £11.25
90th Percentile £32.10 £27.50 £56.12
Central London median hourly rate £20.00 £17.85 £18.45
% change year-on-year +12.04% -3.25% +2.61%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 20 November 2018.

Contractor daily rate histogram for Apache Spark in the City of London

Apache Spark
Contract Job Locations in the City of London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the City of London region over the 6 months to 20 November 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London Bridge - 52 £550 -
Apache Spark
Central London

For the 6 months to 20 November 2018, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 124 (56.11%) Python
2 114 (51.58%) Hadoop
3 108 (48.87%) Kafka
4 98 (44.34%) Big Data
5 93 (42.08%) Finance
6 79 (35.75%) Java
7 67 (30.32%) Elasticsearch
8 63 (28.51%) Agile Software Development
9 58 (26.24%) Amazon AWS
10 56 (25.34%) SQL
10 56 (25.34%) Data Science
11 50 (22.62%) PySpark
11 50 (22.62%) Pandas
12 49 (22.17%) Apache Hive
13 48 (21.72%) NumPy
13 48 (21.72%) Scala
14 45 (20.36%) Docker
15 43 (19.46%) Linux
16 42 (19.00%) Foreign Exchange (FX)
16 42 (19.00%) Splunk
17 39 (17.65%) Machine Learning
18 36 (16.29%) Microservices
19 33 (14.93%) Kubernetes
20 32 (14.48%) Git (software)
21 30 (13.57%) Apache Cassandra
21 30 (13.57%) Data Modelling
22 28 (12.67%) Analytics
23 27 (12.22%) DevOps
23 27 (12.22%) Java 8
23 27 (12.22%) ETL

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 67 (30.32%) Elasticsearch
2 26 (11.76%) Apache
3 8 (3.62%) Apache Airflow
3 8 (3.62%) Apache Pig
3 8 (3.62%) SAS
4 6 (2.71%) Confluence
5 5 (2.26%) Apache Solr
6 4 (1.81%) Oracle Coherence
6 4 (1.81%) Oracle SOA Suite
7 2 (0.90%) Jupyter
8 1 (0.45%) Luigi
8 1 (0.45%) SharePoint
8 1 (0.45%) WordPress
Applications
1 2 (0.90%) Microsoft Excel
1 2 (0.90%) MS Visio
2 1 (0.45%) Microsoft Office
2 1 (0.45%) Microsoft PowerPoint
2 1 (0.45%) Microsoft Project
Business Applications
1 1 (0.45%) Oracle Financials
Cloud Services
1 58 (26.24%) Amazon AWS
2 21 (9.50%) Microsoft Azure
3 16 (7.24%) Google Cloud Platform
4 15 (6.79%) Heroku
5 6 (2.71%) Amazon EMR
5 6 (2.71%) GitHub
6 4 (1.81%) Amazon Kinesis
6 4 (1.81%) Amazon S3
6 4 (1.81%) AWS Lambda
6 4 (1.81%) Serverless
7 3 (1.36%) IBM Cloud
8 2 (0.90%) Cloud Computing
9 1 (0.45%) Azure Data Factory
9 1 (0.45%) Azure Machine Learning
9 1 (0.45%) Data Lake Analytics
Communications & Networking
1 21 (9.50%) HTTP
2 17 (7.69%) SSL
3 15 (6.79%) DNS
3 15 (6.79%) FTP
3 15 (6.79%) TCP/IP
4 6 (2.71%) Kerberos
5 1 (0.45%) DSL
5 1 (0.45%) Internet
5 1 (0.45%) SNA
Database & Business Intelligence
1 114 (51.58%) Hadoop
2 98 (44.34%) Big Data
3 49 (22.17%) Apache Hive
4 30 (13.57%) Apache Cassandra
5 23 (10.41%) HBase
5 23 (10.41%) MongoDB
6 22 (9.95%) NoSQL
7 21 (9.50%) Data Warehouse
8 18 (8.14%) Relational Database
8 18 (8.14%) SQL Server
9 16 (7.24%) Impala
10 14 (6.33%) Amazon Redshift
11 13 (5.88%) PostgreSQL
12 12 (5.43%) Data Mining
12 12 (5.43%) SQL Server Integration Services
13 11 (4.98%) OLAP
13 11 (4.98%) Power BI
14 9 (4.07%) Enterprise Data Warehouse
14 9 (4.07%) OLAP Cube
14 9 (4.07%) Sqoop
Development Applications
1 32 (14.48%) Git (software)
2 21 (9.50%) GitLab
3 20 (9.05%) Jenkins
4 13 (5.88%) Gradle
4 13 (5.88%) gulp
5 10 (4.52%) JIRA
6 8 (3.62%) Atlassian Bamboo
6 8 (3.62%) JUnit
6 8 (3.62%) Maven
6 8 (3.62%) Subversion
7 7 (3.17%) Cucumber
8 6 (2.71%) FishEye
8 6 (2.71%) jMock
8 6 (2.71%) Travis CI
9 4 (1.81%) Visual Studio Team System
10 3 (1.36%) Artifactory
11 2 (0.90%) EasyMock
11 2 (0.90%) Gatling
11 2 (0.90%) GoCD
11 2 (0.90%) TestNG
General
1 93 (42.08%) Finance
2 14 (6.33%) Banking
3 13 (5.88%) Front Office
4 6 (2.71%) Retail
5 2 (0.90%) Investment Banking
5 2 (0.90%) Marketing
5 2 (0.90%) Prime Brokerage
5 2 (0.90%) Publishing
6 1 (0.45%) Advertising
6 1 (0.45%) Corporate Banking
6 1 (0.45%) Electronics
6 1 (0.45%) Financial Institution
6 1 (0.45%) Games
6 1 (0.45%) Law
Job Titles
1 113 (51.13%) Developer
2 64 (28.96%) Data Engineer
3 48 (21.72%) Python Developer
4 37 (16.74%) Apache Spark Developer
5 35 (15.84%) Java Developer
6 24 (10.86%) Big Data Engineer
6 24 (10.86%) Senior Developer
7 22 (9.95%) Elasticsearch Developer
8 17 (7.69%) Python Engineer
8 17 (7.69%) Senior Java Developer
9 15 (6.79%) Big Data Developer
10 13 (5.88%) Data Scientist
11 12 (5.43%) Hadoop Engineer
11 12 (5.43%) Scala Developer
12 10 (4.52%) DevOps Engineer
12 10 (4.52%) Senior Data Engineer
12 10 (4.52%) Technical Leader
13 9 (4.07%) Lead Developer
13 9 (4.07%) Lead Java Developer
13 9 (4.07%) Technical Developer
Libraries, Frameworks & Software Standards
1 108 (48.87%) Kafka
2 50 (22.62%) Pandas
2 50 (22.62%) PySpark
3 48 (21.72%) NumPy
4 19 (8.60%) Spring
5 17 (7.69%) AngularJS
5 17 (7.69%) REST
6 16 (7.24%) CSS
6 16 (7.24%) HTML
6 16 (7.24%) Spark Streaming
7 15 (6.79%) HTML5
7 15 (6.79%) OAuth
7 15 (6.79%) OAuth2
7 15 (6.79%) SOAP
8 13 (5.88%) Dropwizard
8 13 (5.88%) Ember.js
8 13 (5.88%) Guice
8 13 (5.88%) Ratpack
8 13 (5.88%) Spring Boot
9 8 (3.62%) scikit-learn
Miscellaneous
1 42 (19.00%) Foreign Exchange (FX)
2 21 (9.50%) Algorithms
3 17 (7.69%) Greenfield Project
4 13 (5.88%) Fintech
5 5 (2.26%) Computer Science
6 4 (1.81%) Analytical Skills
6 4 (1.81%) Market Risk
7 3 (1.36%) Data Structures
7 3 (1.36%) Derivative
8 2 (0.90%) Cloud Native
8 2 (0.90%) Credit Risk
8 2 (0.90%) Enterprise Software
8 2 (0.90%) Fixed Income
8 2 (0.90%) Mobile App
8 2 (0.90%) OTC Derivatives
8 2 (0.90%) Private Cloud
8 2 (0.90%) Virtual Team
9 1 (0.45%) Equities
9 1 (0.45%) HoloLens
9 1 (0.45%) NHS
Operating Systems
1 43 (19.46%) Linux
2 8 (3.62%) Red Hat Enterprise Linux
2 8 (3.62%) Unix
3 2 (0.90%) Apple iOS
4 1 (0.45%) DC/OS
Processes & Methodologies
1 63 (28.51%) Agile Software Development
2 56 (25.34%) Data Science
3 39 (17.65%) Machine Learning
4 36 (16.29%) Microservices
5 30 (13.57%) Data Modelling
6 28 (12.67%) Analytics
7 27 (12.22%) DevOps
7 27 (12.22%) ETL
8 26 (11.76%) TDD
9 25 (11.31%) Open Source
10 22 (9.95%) Visualisation
11 21 (9.50%) BDD
11 21 (9.50%) Software Engineering
12 19 (8.60%) Business Intelligence
13 18 (8.14%) Artificial Intelligence
13 18 (8.14%) Continuous Integration
14 15 (6.79%) Data Visualisation
14 15 (6.79%) OWASP
14 15 (6.79%) Scrum
15 13 (5.88%) Domain-Driven Design
Programming Languages
1 124 (56.11%) Python
2 79 (35.75%) Java
3 56 (25.34%) SQL
4 48 (21.72%) Scala
5 27 (12.22%) Java 8
6 18 (8.14%) JavaScript
6 18 (8.14%) R
7 15 (6.79%) ES6
7 15 (6.79%) TypeScript
8 9 (4.07%) Bash Shell
9 7 (3.17%) MATLAB
10 3 (1.36%) Clojure
10 3 (1.36%) Kotlin
10 3 (1.36%) Ruby
11 2 (0.90%) C
11 2 (0.90%) C#
11 2 (0.90%) C++
11 2 (0.90%) Data Analysis Expressions (DAX)
11 2 (0.90%) Perl
11 2 (0.90%) T-SQL
Qualifications
1 7 (3.17%) Degree
2 3 (1.36%) ISTQB
2 3 (1.36%) ISTQB Advanced Certification
2 3 (1.36%) PhD
3 2 (0.90%) Master's Degree
3 2 (0.90%) Numerate Degree
4 1 (0.45%) Computer Science Degree
4 1 (0.45%) Microsoft Certification
Quality Assurance & Compliance
1 11 (4.98%) Data Quality
2 7 (3.17%) QA
System Software
1 45 (20.36%) Docker
2 21 (9.50%) HDFS
3 16 (7.24%) Apache Flume
4 5 (2.26%) Active Directory
5 2 (0.90%) Apache ZooKeeper
Systems Management
1 33 (14.93%) Kubernetes
2 18 (8.14%) Oozie
3 17 (7.69%) Hadoop YARN
4 13 (5.88%) Docker Swarm
5 10 (4.52%) Ansible
6 5 (2.26%) Ambari
6 5 (2.26%) Puppet
6 5 (2.26%) Terraform
7 3 (1.36%) ITRS Geneos
7 3 (1.36%) Mesos
8 2 (0.90%) Apache Slider
8 2 (0.90%) Kibana
8 2 (0.90%) logstash
8 2 (0.90%) Marathon
8 2 (0.90%) Opscode Chef
8 2 (0.90%) Prometheus
8 2 (0.90%) rkt
9 1 (0.45%) Tivoli
Vendors
1 42 (19.00%) Splunk
2 26 (11.76%) Oracle
3 15 (6.79%) Cloudera
3 15 (6.79%) Google
4 13 (5.88%) Tableau
5 12 (5.43%) Hortonworks
6 11 (4.98%) Talend
7 9 (4.07%) Informatica
8 8 (3.62%) Sybase
9 6 (2.71%) Alteryx
9 6 (2.71%) Atlassian
9 6 (2.71%) Red Hat
10 5 (2.26%) MapR
11 4 (1.81%) IBM
11 4 (1.81%) Microsoft
12 3 (1.36%) Ab Initio
12 3 (1.36%) TIBCO
13 1 (0.45%) Databricks
13 1 (0.45%) Murex
13 1 (0.45%) Teradata