Period
to 23 January 2019

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 23 January 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
23 Jan 2019
Same period 2018 Same period 2017
Rank 61 97 86
Rank change year-on-year +36 -11 +80
Contract jobs citing Apache Spark 268 181 183
As % of all contract IT jobs advertised in the City of London 3.92% 2.54% 2.97%
As % of the Application Platforms category 21.27% 17.57% 16.22%
Number of daily rates quoted 227 165 159
City of London median daily rate £550 £588 £600
Median daily rate % change year-on-year -6.38% -2.08% +4.35%
10th Percentile £513 £475 £450
90th Percentile £718 £700 £688
Central London median daily rate £550 £575 £600
% change year-on-year -4.35% -4.17% +6.67%
Number of hourly rates quoted 0 0 0
City of London median hourly rate - - -
Central London median hourly rate - - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,260 1,030 1,128
As % of all contract IT jobs advertised in the City of London 18.45% 14.45% 18.28%
Number of daily rates quoted 955 830 909
City of London median daily rate £525 £500 £475
Median daily rate % change year-on-year +5.00% +5.26% +8.57%
10th Percentile £275 £280 £313
90th Percentile £650 £675 £658
Central London median daily rate £525 £498 £465
% change year-on-year +5.53% +6.99% +9.41%
Number of hourly rates quoted 22 10 11
City of London median hourly rate £20.88 £17.85 £19.00
Median hourly rate % change year-on-year +16.95% -6.05% -2.56%
10th Percentile £13.00 £11.13 £11.25
90th Percentile £32.93 £27.50 £56.87
Central London median hourly rate £19.25 £17.85 £18.45
% change year-on-year +7.84% -3.25% +2.50%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 23 January 2019.

Contractor daily rate histogram for Apache Spark in the City of London

Apache Spark
Contract Job Locations in the City of London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the City of London region over the 6 months to 23 January 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London Bridge - 100 £525 -
Apache Spark
Central London

For the 6 months to 23 January 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 146 (54.48%) Python
2 140 (52.24%) Finance
3 131 (48.88%) Hadoop
4 130 (48.51%) Big Data
5 122 (45.52%) Kafka
6 91 (33.96%) Data Science
7 84 (31.34%) PySpark
8 83 (30.97%) Java
9 80 (29.85%) Elasticsearch
10 73 (27.24%) SQL
11 71 (26.49%) Machine Learning
12 67 (25.00%) Apache Hive
12 67 (25.00%) Pandas
13 65 (24.25%) NumPy
14 63 (23.51%) Foreign Exchange (FX)
15 61 (22.76%) Splunk
16 57 (21.27%) Data Modelling
16 57 (21.27%) Amazon AWS
17 53 (19.78%) Agile Software Development
18 51 (19.03%) Analytics
19 45 (16.79%) Scala
20 42 (15.67%) Data Warehouse
21 37 (13.81%) Docker
22 36 (13.43%) Apache
23 35 (13.06%) Linux
24 34 (12.69%) OLAP
24 34 (12.69%) Oozie
25 33 (12.31%) Microservices
26 32 (11.94%) Data Quality
27 30 (11.19%) SQL Optimisation

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 80 (29.85%) Elasticsearch
2 36 (13.43%) Apache
3 12 (4.48%) Apache Airflow
4 10 (3.73%) Confluence
5 5 (1.87%) SAS
6 4 (1.49%) Jupyter
6 4 (1.49%) Oracle Coherence
6 4 (1.49%) Oracle SOA Suite
7 3 (1.12%) Apache Pig
8 1 (0.37%) Apache Solr
8 1 (0.37%) Luigi
8 1 (0.37%) SharePoint
8 1 (0.37%) WordPress
Applications
1 2 (0.75%) Microsoft Excel
2 1 (0.37%) Microsoft Project
2 1 (0.37%) MS Visio
Business Applications
1 1 (0.37%) Dynamics NAV
1 1 (0.37%) Oracle Financials
Cloud Services
1 57 (21.27%) Amazon AWS
2 26 (9.70%) Microsoft Azure
3 22 (8.21%) Google Cloud Platform
4 12 (4.48%) Heroku
5 5 (1.87%) GitHub
6 3 (1.12%) Amazon EMR
6 3 (1.12%) IBM Cloud
7 2 (0.75%) Amazon Kinesis
7 2 (0.75%) Apigee
7 2 (0.75%) Serverless
8 1 (0.37%) AWS Lambda
Communications & Networking
1 19 (7.09%) HTTP
2 13 (4.85%) DNS
3 12 (4.48%) FTP
3 12 (4.48%) SSL
3 12 (4.48%) TCP/IP
4 4 (1.49%) Kerberos
5 1 (0.37%) DSL
5 1 (0.37%) Internet
Database & Business Intelligence
1 131 (48.88%) Hadoop
2 130 (48.51%) Big Data
3 67 (25.00%) Apache Hive
4 42 (15.67%) Data Warehouse
5 34 (12.69%) OLAP
6 30 (11.19%) Enterprise Data Warehouse
6 30 (11.19%) OLAP Cube
6 30 (11.19%) Star Schema
7 23 (8.58%) Apache Cassandra
8 21 (7.84%) Impala
9 20 (7.46%) SQL Server
10 19 (7.09%) MongoDB
11 18 (6.72%) Relational Database
12 16 (5.97%) NoSQL
13 15 (5.60%) HBase
14 11 (4.10%) Apache Kudu
14 11 (4.10%) Data Mining
15 10 (3.73%) Data Lake
16 9 (3.36%) Power BI
16 9 (3.36%) SQL Server Integration Services
Development Applications
1 29 (10.82%) Git (software)
2 20 (7.46%) Jenkins
3 14 (5.22%) GitLab
3 14 (5.22%) Maven
4 13 (4.85%) Cucumber
4 13 (4.85%) JIRA
5 12 (4.48%) Atlassian Bamboo
5 12 (4.48%) Gradle
5 12 (4.48%) gulp
5 12 (4.48%) JUnit
5 12 (4.48%) Subversion
6 10 (3.73%) FishEye
6 10 (3.73%) jMock
7 4 (1.49%) Visual Studio Team System
8 3 (1.12%) Artifactory
8 3 (1.12%) Visual Studio
9 2 (0.75%) EasyMock
9 2 (0.75%) Gatling
9 2 (0.75%) Mockito
9 2 (0.75%) TestNG
General
1 140 (52.24%) Finance
2 24 (8.96%) Front Office
3 23 (8.58%) Banking
4 11 (4.10%) Retail
5 5 (1.87%) Investment Banking
6 2 (0.75%) Financial Institution
6 2 (0.75%) Marketing
7 1 (0.37%) Corporate Banking
7 1 (0.37%) Electronics
7 1 (0.37%) Law
7 1 (0.37%) Publishing
Job Titles
1 123 (45.90%) Developer
2 88 (32.84%) Data Engineer
3 59 (22.01%) Python Developer
4 44 (16.42%) Apache Spark Developer
5 37 (13.81%) Java Developer
6 32 (11.94%) Hadoop Engineer
7 31 (11.57%) Big Data Engineer
8 23 (8.58%) Data Scientist
9 22 (8.21%) Elasticsearch Developer
10 20 (7.46%) Python Engineer
10 20 (7.46%) Senior Developer
11 19 (7.09%) Senior Java Developer
12 17 (6.34%) Scala Developer
13 13 (4.85%) Big Data Developer
14 12 (4.48%) Technical Leader
15 10 (3.73%) DevOps Engineer
16 9 (3.36%) Senior Data Engineer
17 8 (2.99%) Lead Developer
17 8 (2.99%) Lead Java Developer
17 8 (2.99%) Technical Developer
Libraries, Frameworks & Software Standards
1 122 (45.52%) Kafka
2 84 (31.34%) PySpark
3 67 (25.00%) Pandas
4 65 (24.25%) NumPy
5 22 (8.21%) Spring
6 14 (5.22%) Spring Boot
7 13 (4.85%) REST
8 12 (4.48%) AngularJS
8 12 (4.48%) CSS
8 12 (4.48%) Dropwizard
8 12 (4.48%) Ember.js
8 12 (4.48%) Guice
8 12 (4.48%) HTML
8 12 (4.48%) HTML5
8 12 (4.48%) OAuth
8 12 (4.48%) OAuth2
8 12 (4.48%) Ratpack
8 12 (4.48%) SOAP
8 12 (4.48%) Spark Streaming
9 11 (4.10%) Akka
Miscellaneous
1 63 (23.51%) Foreign Exchange (FX)
2 27 (10.07%) Algorithms
3 16 (5.97%) Greenfield Project
4 12 (4.48%) Fintech
5 8 (2.99%) Data Structures
6 6 (2.24%) Analytical Skills
6 6 (2.24%) Market Risk
7 2 (0.75%) Mobile App
7 2 (0.75%) Private Cloud
7 2 (0.75%) Virtual Team
8 1 (0.37%) Cloud Native
8 1 (0.37%) Derivative
8 1 (0.37%) HoloLens
8 1 (0.37%) Internet of Things
8 1 (0.37%) NHS
Operating Systems
1 35 (13.06%) Linux
2 11 (4.10%) Unix
3 5 (1.87%) Red Hat Enterprise Linux
4 2 (0.75%) Apple iOS
Processes & Methodologies
1 91 (33.96%) Data Science
2 71 (26.49%) Machine Learning
3 57 (21.27%) Data Modelling
4 53 (19.78%) Agile Software Development
5 51 (19.03%) Analytics
6 33 (12.31%) Microservices
7 30 (11.19%) SQL Optimisation
8 26 (9.70%) DevOps
9 24 (8.96%) Data Engineering
10 23 (8.58%) BDD
10 23 (8.58%) TDD
11 22 (8.21%) Business Intelligence
12 20 (7.46%) Open Source
12 20 (7.46%) Visualisation
13 19 (7.09%) ETL
14 18 (6.72%) Artificial Intelligence
15 17 (6.34%) Problem-Solving
15 17 (6.34%) Software Engineering
16 16 (5.97%) SDLC
17 15 (5.60%) Mathematics
Programming Languages
1 146 (54.48%) Python
2 83 (30.97%) Java
3 73 (27.24%) SQL
4 45 (16.79%) Scala
5 29 (10.82%) Java 8
6 15 (5.60%) JavaScript
7 14 (5.22%) R
8 12 (4.48%) ES6
8 12 (4.48%) TypeScript
9 10 (3.73%) Bash Shell
10 5 (1.87%) MATLAB
11 3 (1.12%) Clojure
11 3 (1.12%) Data Analysis Expressions (DAX)
11 3 (1.12%) Kotlin
12 2 (0.75%) C#
12 2 (0.75%) Cypher
12 2 (0.75%) Shell Script
12 2 (0.75%) T-SQL
13 1 (0.37%) Ruby
13 1 (0.37%) VBA
Qualifications
1 12 (4.48%) Degree
2 9 (3.36%) PhD
3 5 (1.87%) Master's Degree
3 5 (1.87%) Numerate Degree
4 3 (1.12%) ISTQB
4 3 (1.12%) ISTQB Advanced Certification
5 1 (0.37%) Computer Science Degree
Quality Assurance & Compliance
1 32 (11.94%) Data Quality
2 7 (2.61%) QA
System Software
1 37 (13.81%) Docker
2 19 (7.09%) HDFS
3 14 (5.22%) Apache Flume
4 3 (1.12%) Active Directory
Systems Management
1 34 (12.69%) Oozie
2 30 (11.19%) Kubernetes
3 13 (4.85%) Ansible
4 12 (4.48%) Docker Swarm
4 12 (4.48%) Hadoop YARN
5 6 (2.24%) Puppet
6 4 (1.49%) Terraform
7 3 (1.12%) Ambari
7 3 (1.12%) ITRS Geneos
8 2 (0.75%) Opscode Chef
9 1 (0.37%) Kibana
9 1 (0.37%) logstash
9 1 (0.37%) rkt
9 1 (0.37%) Tivoli
Vendors
1 61 (22.76%) Splunk
2 26 (9.70%) Oracle
3 22 (8.21%) Cloudera
4 19 (7.09%) Google
5 13 (4.85%) Talend
6 11 (4.10%) Hortonworks
7 10 (3.73%) Atlassian
7 10 (3.73%) Sybase
7 10 (3.73%) Tableau
8 5 (1.87%) Microsoft
8 5 (1.87%) Red Hat
9 4 (1.49%) Alteryx
9 4 (1.49%) Informatica
10 3 (1.12%) IBM
10 3 (1.12%) MapR
11 1 (0.37%) Databricks
11 1 (0.37%) Murex