Period
to 26 June 2019

The following table provides summary statistics for contract job vacancies advertised in Central London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 26 June 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
London > Central London
6 months to
26 Jun 2019
Same period 2018 Same period 2017
Rank 62 116 122
Rank change year-on-year +54 +6 +63
Contract jobs citing Apache Spark 339 216 146
As % of all contract IT jobs advertised in Central London 4.01% 2.37% 1.97%
As % of the Application Platforms category 23.24% 13.91% 11.35%
Number of daily rates quoted 297 195 126
Central London median daily rate £550 £550 £600
Median daily rate % change year-on-year - -8.33% +6.67%
10th Percentile £513 £463 £450
90th Percentile £688 £688 £688
London median daily rate £575 £550 £550
% change year-on-year +4.55% - -4.35%
Number of hourly rates quoted 0 0 0
Central London median hourly rate - - -
London median hourly rate - £15.50 -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in Central London with a requirement for application platform skills.

All Application Platform Skills
Central London
Contract vacancies with a requirement for application platform skills 1,459 1,553 1,286
As % of all contract IT jobs advertised in Central London 17.26% 17.02% 17.34%
Number of daily rates quoted 1,099 1,195 952
Central London median daily rate £525 £500 £475
Median daily rate % change year-on-year +5.00% +5.26% +5.56%
10th Percentile £331 £300 £269
90th Percentile £673 £638 £638
London median daily rate £525 £500 £475
% change year-on-year +5.00% +5.26% +5.56%
Number of hourly rates quoted 13 25 6
Central London median hourly rate £20.00 £20.00 £26.17
Median hourly rate % change year-on-year - -23.57% +37.72%
10th Percentile £14.11 £12.80 £12.88
90th Percentile £37.00 £33.45 £83.13
London median hourly rate £28.50 £20.88 £22.17
% change year-on-year +36.53% -5.83% +11.31%

Apache Spark
Job Vacancy Trend in Central London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in Central London.

Job vacancy trend for Apache Spark in Central London

Apache Spark
Contractor Daily Rate Trend in Central London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in Central London.

Contractor daily rate trend for Apache Spark in Central London

Apache Spark
Contractor Daily Rate Histogram in Central London

The daily rate distribution of IT jobs citing Apache Spark in Central London over the 6 months to 26 June 2019.

Contractor daily rate histogram for Apache Spark in Central London

Apache Spark
Contract Job Locations in Central London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the Central London region over the 6 months to 26 June 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
City of London +53 270 £550 -4.35% 20
Southwark +18 114 £525 -8.70%
West End of London +8 7 £425 -15.00%
City of Westminster -17 3 £600 +9.09%
Apache Spark
London

For the 6 months to 26 June 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the Central London region with a requirement for Apache Spark.

1 247 (72.86%) Big Data
2 201 (59.29%) Python
3 189 (55.75%) Finance
4 186 (54.87%) Hadoop
5 151 (44.54%) Data Science
5 151 (44.54%) Kafka
6 145 (42.77%) Apache Hive
7 136 (40.12%) SQL
8 134 (39.53%) Amazon AWS
9 101 (29.79%) Data Modelling
10 94 (27.73%) PySpark
11 93 (27.43%) Data Warehouse
12 90 (26.55%) Agile Software Development
13 89 (26.25%) Google
13 89 (26.25%) Google Cloud Platform
14 88 (25.96%) Machine Learning
15 83 (24.48%) Analytics
16 81 (23.89%) Microsoft Azure
17 80 (23.60%) Data Engineering
18 75 (22.12%) Java
19 65 (19.17%) Apache
20 64 (18.88%) Scala
21 62 (18.29%) DevOps
22 56 (16.52%) Apache Airflow
23 52 (15.34%) Mentoring
24 51 (15.04%) Cloudera
25 50 (14.75%) Coaching
25 50 (14.75%) Data Cleansing
26 44 (12.98%) Data Quality
26 44 (12.98%) OLAP

Apache Spark
Co-occurring IT Skills in Central London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 65 (19.17%) Apache
2 56 (16.52%) Apache Airflow
3 32 (9.44%) Elasticsearch
4 9 (2.65%) Confluence
5 8 (2.36%) Apache Solr
6 5 (1.47%) Pachyderm
7 4 (1.18%) Jupyter
8 2 (0.59%) ArcGIS
8 2 (0.59%) SAS
9 1 (0.29%) Apache Pig
9 1 (0.29%) SharePoint
Applications
1 20 (5.90%) Microsoft Excel
Business Applications
1 7 (2.06%) SAP Oil and Gas
2 2 (0.59%) SAP ERP
2 2 (0.59%) SAP S/4HANA
Cloud Services
1 134 (39.53%) Amazon AWS
2 89 (26.25%) Google Cloud Platform
3 81 (23.89%) Microsoft Azure
4 16 (4.72%) Serverless
5 12 (3.54%) Amazon EMR
5 12 (3.54%) Power Platform
6 10 (2.95%) Amazon S3
6 10 (2.95%) Amazon SageMaker
6 10 (2.95%) AWS Lambda
7 8 (2.36%) Amazon Kinesis
8 6 (1.77%) Azure Data Factory
8 6 (1.77%) GitHub
9 5 (1.47%) PaaS
10 4 (1.18%) Amazon EC2
10 4 (1.18%) Apigee
10 4 (1.18%) AWS CloudFormation
11 3 (0.88%) Azure Functions
11 3 (0.88%) Cloud Computing
11 3 (0.88%) Data Lake Analytics
11 3 (0.88%) SaaS
Communications & Networking
1 4 (1.18%) DNS
2 3 (0.88%) HTTP
3 2 (0.59%) FTP
3 2 (0.59%) SSL
3 2 (0.59%) TCP/IP
4 1 (0.29%) Firewall
4 1 (0.29%) Kerberos
Database & Business Intelligence
1 247 (72.86%) Big Data
2 186 (54.87%) Hadoop
3 145 (42.77%) Apache Hive
4 93 (27.43%) Data Warehouse
5 44 (12.98%) OLAP
6 42 (12.39%) Star Schema
7 41 (12.09%) Data Lake
7 41 (12.09%) Enterprise Data Warehouse
7 41 (12.09%) OLAP Cube
8 38 (11.21%) HBase
9 37 (10.91%) Impala
10 33 (9.73%) NoSQL
11 32 (9.44%) SQL Server
12 20 (5.90%) MongoDB
13 17 (5.01%) Apache Cassandra
14 14 (4.13%) Amazon Redshift
15 13 (3.83%) Sqoop
16 12 (3.54%) PostgreSQL
16 12 (3.54%) Power BI
17 10 (2.95%) Spark SQL
Development Applications
1 26 (7.67%) Jenkins
2 21 (6.19%) JIRA
3 19 (5.60%) Git (software)
4 7 (2.06%) IBM UrbanCode
4 7 (2.06%) RStudio
4 7 (2.06%) Visual Studio Team System
5 6 (1.77%) Maven
5 6 (1.77%) Travis CI
6 5 (1.47%) Cucumber
7 4 (1.18%) Subversion
8 3 (0.88%) GitLab
8 3 (0.88%) Octopus Deploy
8 3 (0.88%) SpecFlow
8 3 (0.88%) TeamCity
8 3 (0.88%) tSQLt
9 2 (0.59%) Atlassian Bamboo
9 2 (0.59%) jMock
9 2 (0.59%) JUnit
9 2 (0.59%) Sonatype Nexus
9 2 (0.59%) Visual Studio
General
1 189 (55.75%) Finance
2 37 (10.91%) Banking
3 11 (3.24%) Retail
4 10 (2.95%) Front Office
4 10 (2.95%) Investment Banking
5 6 (1.77%) Retail Banking
6 2 (0.59%) Back Office
6 2 (0.59%) Publishing
7 1 (0.29%) Pharmaceutical
7 1 (0.29%) Telecoms
Job Titles
1 200 (59.00%) Data Engineer
2 110 (32.45%) Big Data Engineer
3 77 (22.71%) Python Engineer
4 66 (19.47%) Developer
5 53 (15.63%) Hadoop Engineer
6 25 (7.37%) Python Developer
7 24 (7.08%) Cloud Engineer
8 23 (6.78%) Java Developer
9 17 (5.01%) Big Data Developer
10 15 (4.42%) Data Scientist
11 11 (3.24%) AWS Engineer
11 11 (3.24%) Senior Developer
12 10 (2.95%) Apache Spark Developer
12 10 (2.95%) DevOps Engineer
13 9 (2.65%) Architect
13 9 (2.65%) Scala Developer
14 8 (2.36%) Hadoop Developer
14 8 (2.36%) Java Engineer
15 7 (2.06%) Java Hadoop Developer
15 7 (2.06%) Platform Engineer
Libraries, Frameworks & Software Standards
1 151 (44.54%) Kafka
2 94 (27.73%) PySpark
3 31 (9.14%) Pandas
4 25 (7.37%) NumPy
5 24 (7.08%) Spring
6 18 (5.31%) scikit-learn
7 12 (3.54%) Spark Streaming
7 12 (3.54%) Spring Boot
8 11 (3.24%) Play Framework
9 10 (2.95%) MLlib
9 10 (2.95%) RESTful
10 9 (2.65%) Natural Language Processing
10 9 (2.65%) Web Services
11 8 (2.36%) TensorFlow
12 6 (1.77%) Apache NiFi
13 5 (1.47%) Elastic Stack
13 5 (1.47%) RabbitMQ
13 5 (1.47%) React
13 5 (1.47%) Velocity
14 4 (1.18%) .NET
Miscellaneous
1 24 (7.08%) Algorithms
2 21 (6.19%) Foreign Exchange (FX)
3 19 (5.60%) Data Structures
4 15 (4.42%) Analytical Skills
4 15 (4.42%) Greenfield Project
5 7 (2.06%) Linux Command Line
6 4 (1.18%) Cloud Native
6 4 (1.18%) Life Science
7 3 (0.88%) Enterprise Software
7 3 (0.88%) Self-Motivation
8 2 (0.59%) Data Centre
8 2 (0.59%) Hedge funds
8 2 (0.59%) Market Risk
8 2 (0.59%) Public Cloud
8 2 (0.59%) User Experience
9 1 (0.29%) NHS
9 1 (0.29%) Social Media
9 1 (0.29%) Virtual Team
Operating Systems
1 40 (11.80%) Linux
2 29 (8.55%) Unix
3 5 (1.47%) Windows
4 2 (0.59%) Red Hat Enterprise Linux
Processes & Methodologies
1 151 (44.54%) Data Science
2 101 (29.79%) Data Modelling
3 90 (26.55%) Agile Software Development
4 88 (25.96%) Machine Learning
5 83 (24.48%) Analytics
6 80 (23.60%) Data Engineering
7 62 (18.29%) DevOps
8 52 (15.34%) Mentoring
9 50 (14.75%) Coaching
9 50 (14.75%) Data Cleansing
10 41 (12.09%) SQL Optimisation
11 35 (10.32%) Scrum
12 31 (9.14%) ETL
12 31 (9.14%) TDD
13 28 (8.26%) Single Customer View
14 27 (7.96%) SDLC
15 26 (7.67%) Artificial Intelligence
16 25 (7.37%) Performance Testing
17 23 (6.78%) Data Analysis
17 23 (6.78%) Migration
Programming Languages
1 201 (59.29%) Python
2 136 (40.12%) SQL
3 75 (22.12%) Java
4 64 (18.88%) Scala
5 23 (6.78%) R
6 14 (4.13%) Java 8
7 9 (2.65%) JavaScript
8 7 (2.06%) Shell Script
9 5 (1.47%) Bash Shell
9 5 (1.47%) C#
9 5 (1.47%) Clojure
9 5 (1.47%) Julia
9 5 (1.47%) U-SQL
10 4 (1.18%) Kotlin
10 4 (1.18%) T-SQL
11 2 (0.59%) ES6
11 2 (0.59%) MATLAB
11 2 (0.59%) Perl
11 2 (0.59%) PowerShell
11 2 (0.59%) VBA
Qualifications
1 18 (5.31%) PMI Certification
1 18 (5.31%) PMP
2 13 (3.83%) Degree
3 7 (2.06%) PhD
4 5 (1.47%) Numerate Degree
4 5 (1.47%) SC Cleared
4 5 (1.47%) Security Cleared
5 2 (0.59%) Master's Degree
6 1 (0.29%) CCA
6 1 (0.29%) Computer Science Degree
6 1 (0.29%) Postgraduate
Quality Assurance & Compliance
1 44 (12.98%) Data Quality
2 6 (1.77%) GDPR
3 1 (0.29%) QA
System Software
1 41 (12.09%) Docker
2 32 (9.44%) Apache Flume
3 23 (6.78%) HDFS
4 1 (0.29%) Active Directory
4 1 (0.29%) Apache ZooKeeper
Systems Management
1 43 (12.68%) Oozie
2 23 (6.78%) Hadoop YARN
2 23 (6.78%) Kubernetes
3 18 (5.31%) Ansible
4 7 (2.06%) Terraform
5 6 (1.77%) Kubeflow
6 4 (1.18%) Kibana
6 4 (1.18%) Puppet
7 3 (0.88%) logstash
7 3 (0.88%) Marathon
7 3 (0.88%) Mesos
8 2 (0.59%) Docker Swarm
9 1 (0.29%) Grafana
9 1 (0.29%) Istio
9 1 (0.29%) rkt
9 1 (0.29%) Tivoli
Vendors
1 89 (26.25%) Google
2 51 (15.04%) Cloudera
3 35 (10.32%) Talend
4 32 (9.44%) Oracle
5 26 (7.67%) Informatica
6 18 (5.31%) Splunk
7 14 (4.13%) Tableau
8 10 (2.95%) Microsoft
9 7 (2.06%) Databricks
9 7 (2.06%) SAP
10 5 (1.47%) Hortonworks
11 4 (1.18%) StreamSets
11 4 (1.18%) Teradata
12 2 (0.59%) Alteryx
12 2 (0.59%) Atlassian
12 2 (0.59%) Datameer
12 2 (0.59%) Sybase
13 1 (0.29%) MapR