Period
to 17 June 2019

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 17 June 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
17 Jun 2019
Same period 2018 Same period 2017
Rank 49 111 110
Rank change year-on-year +62 -1 +66
Contract jobs citing Apache Spark 281 162 134
As % of all contract IT jobs advertised in the City of London 4.38% 2.35% 2.16%
As % of the Application Platforms category 24.69% 14.58% 12.37%
Number of daily rates quoted 249 143 115
City of London median daily rate £550 £575 £600
Median daily rate % change year-on-year -4.35% -4.17% +9.09%
10th Percentile £513 £463 £450
90th Percentile £688 £736 £691
Central London median daily rate £550 £550 £600
% change year-on-year - -8.33% +6.67%
Number of hourly rates quoted 0 0 0
City of London median hourly rate - - -
Central London median hourly rate - - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,138 1,111 1,083
As % of all contract IT jobs advertised in the City of London 17.75% 16.12% 17.43%
Number of daily rates quoted 878 870 820
City of London median daily rate £525 £500 £475
Median daily rate % change year-on-year +5.00% +5.26% +5.56%
10th Percentile £313 £310 £275
90th Percentile £675 £650 £638
Central London median daily rate £525 £500 £475
% change year-on-year +5.00% +5.26% +5.56%
Number of hourly rates quoted 9 19 5
City of London median hourly rate £19.50 £20.00 £30.00
Median hourly rate % change year-on-year -2.50% -33.33% +57.89%
10th Percentile £12.48 £12.50 £16.39
90th Percentile £42.00 £35.75 £92.25
Central London median hourly rate £19.75 £20.00 £26.17
% change year-on-year -1.25% -23.57% +37.72%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 17 June 2019.

Contractor daily rate histogram for Apache Spark in the City of London

Apache Spark
Contract Job Locations in the City of London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the City of London region over the 6 months to 17 June 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London Bridge +14 126 £525 -8.70%
Moorgate - 10 £575 -
Apache Spark
Central London

For the 6 months to 17 June 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 186 (66.19%) Big Data
2 167 (59.43%) Python
3 165 (58.72%) Finance
4 149 (53.02%) Data Science
5 140 (49.82%) Hadoop
6 121 (43.06%) Amazon AWS
7 119 (42.35%) Kafka
8 110 (39.15%) SQL
9 99 (35.23%) Apache Hive
10 97 (34.52%) PySpark
11 87 (30.96%) Machine Learning
12 83 (29.54%) Google Cloud Platform
13 82 (29.18%) Analytics
14 79 (28.11%) Google
15 76 (27.05%) Data Modelling
16 72 (25.62%) Microsoft Azure
17 70 (24.91%) Java
18 69 (24.56%) Apache
19 63 (22.42%) Agile Software Development
20 62 (22.06%) Data Engineering
21 61 (21.71%) Scala
22 60 (21.35%) Data Warehouse
22 60 (21.35%) Apache Airflow
23 56 (19.93%) Mentoring
24 54 (19.22%) Coaching
24 54 (19.22%) Data Cleansing
25 48 (17.08%) OLAP
26 46 (16.37%) Star Schema
26 46 (16.37%) Oozie
26 46 (16.37%) Data Quality

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 69 (24.56%) Apache
2 60 (21.35%) Apache Airflow
3 34 (12.10%) Elasticsearch
4 10 (3.56%) Confluence
5 7 (2.49%) Apache Solr
6 4 (1.42%) Jupyter
7 1 (0.36%) Apache Pig
7 1 (0.36%) ArcGIS
7 1 (0.36%) SharePoint
Applications
1 2 (0.71%) Microsoft Excel
Business Applications
1 2 (0.71%) SAP ERP
1 2 (0.71%) SAP Oil and Gas
1 2 (0.71%) SAP S/4HANA
Cloud Services
1 121 (43.06%) Amazon AWS
2 83 (29.54%) Google Cloud Platform
3 72 (25.62%) Microsoft Azure
4 15 (5.34%) Serverless
5 11 (3.91%) Power Platform
6 10 (3.56%) Amazon SageMaker
7 9 (3.20%) Amazon Kinesis
7 9 (3.20%) Amazon S3
7 9 (3.20%) AWS Lambda
8 6 (2.14%) Amazon EMR
8 6 (2.14%) Azure Data Factory
8 6 (2.14%) GitHub
9 4 (1.42%) Apigee
10 3 (1.07%) AWS CloudFormation
10 3 (1.07%) Azure Functions
10 3 (1.07%) Cloud Computing
10 3 (1.07%) Data Lake Analytics
10 3 (1.07%) Datadog
10 3 (1.07%) PaaS
10 3 (1.07%) SaaS
Communications & Networking
1 4 (1.42%) DNS
2 3 (1.07%) HTTP
3 2 (0.71%) FTP
3 2 (0.71%) SSL
3 2 (0.71%) TCP/IP
4 1 (0.36%) Firewall
4 1 (0.36%) Kerberos
Database & Business Intelligence
1 186 (66.19%) Big Data
2 140 (49.82%) Hadoop
3 99 (35.23%) Apache Hive
4 60 (21.35%) Data Warehouse
5 48 (17.08%) OLAP
6 46 (16.37%) Star Schema
7 45 (16.01%) Enterprise Data Warehouse
7 45 (16.01%) OLAP Cube
8 34 (12.10%) HBase
9 28 (9.96%) NoSQL
10 18 (6.41%) Impala
11 17 (6.05%) MongoDB
12 14 (4.98%) SQL Server
13 13 (4.63%) Data Lake
14 12 (4.27%) Apache Cassandra
15 11 (3.91%) Power BI
16 10 (3.56%) PostgreSQL
16 10 (3.56%) Spark SQL
17 9 (3.20%) Sqoop
18 8 (2.85%) MySQL
Development Applications
1 27 (9.61%) Jenkins
2 22 (7.83%) JIRA
3 21 (7.47%) Git (software)
4 7 (2.49%) IBM UrbanCode
4 7 (2.49%) Maven
4 7 (2.49%) RStudio
4 7 (2.49%) Visual Studio Team System
5 6 (2.14%) Cucumber
5 6 (2.14%) Travis CI
6 5 (1.78%) Subversion
7 3 (1.07%) Atlassian Bamboo
7 3 (1.07%) FishEye
7 3 (1.07%) jMock
7 3 (1.07%) JUnit
7 3 (1.07%) Octopus Deploy
7 3 (1.07%) SpecFlow
7 3 (1.07%) TeamCity
7 3 (1.07%) tSQLt
8 2 (0.71%) GitLab
8 2 (0.71%) Sonatype Nexus
General
1 165 (58.72%) Finance
2 35 (12.46%) Banking
3 12 (4.27%) Retail
4 11 (3.91%) Front Office
5 9 (3.20%) Investment Banking
6 6 (2.14%) Retail Banking
7 2 (0.71%) Back Office
7 2 (0.71%) Publishing
8 1 (0.36%) Pharmaceutical
8 1 (0.36%) Telecoms
Job Titles
1 149 (53.02%) Data Engineer
2 66 (23.49%) Python Engineer
3 65 (23.13%) Developer
4 64 (22.78%) Big Data Engineer
5 54 (19.22%) Hadoop Engineer
6 29 (10.32%) Python Developer
7 22 (7.83%) Java Developer
8 14 (4.98%) Big Data Developer
9 12 (4.27%) Data Scientist
10 11 (3.91%) Apache Spark Developer
11 10 (3.56%) DevOps Engineer
12 8 (2.85%) Java Engineer
12 8 (2.85%) Scala Developer
12 8 (2.85%) Senior Developer
13 7 (2.49%) AWS Engineer
13 7 (2.49%) Hadoop Developer
13 7 (2.49%) Platform Engineer
13 7 (2.49%) Technical Leader
13 7 (2.49%) Unix Engineer
14 6 (2.14%) AWS Developer
Libraries, Frameworks & Software Standards
1 119 (42.35%) Kafka
2 97 (34.52%) PySpark
3 28 (9.96%) NumPy
3 28 (9.96%) Pandas
4 23 (8.19%) Spring
5 17 (6.05%) scikit-learn
6 12 (4.27%) Spring Boot
7 11 (3.91%) Play Framework
8 10 (3.56%) MLlib
9 9 (3.20%) Natural Language Processing
9 9 (3.20%) RESTful
9 9 (3.20%) Spark Streaming
10 8 (2.85%) Web Services
11 6 (2.14%) Apache NiFi
11 6 (2.14%) TensorFlow
12 5 (1.78%) Elastic Stack
12 5 (1.78%) React
12 5 (1.78%) Velocity
13 4 (1.42%) .NET
13 4 (1.42%) REST
Miscellaneous
1 24 (8.54%) Foreign Exchange (FX)
2 20 (7.12%) Algorithms
3 19 (6.76%) Data Structures
4 14 (4.98%) Greenfield Project
5 13 (4.63%) Analytical Skills
6 7 (2.49%) Linux Command Line
7 3 (1.07%) Cloud Native
7 3 (1.07%) Enterprise Software
8 2 (0.71%) Market Risk
9 1 (0.36%) Data Centre
9 1 (0.36%) NHS
9 1 (0.36%) Public Cloud
9 1 (0.36%) Self-Motivation
9 1 (0.36%) Social Media
9 1 (0.36%) User Experience
Operating Systems
1 36 (12.81%) Linux
2 28 (9.96%) Unix
3 2 (0.71%) Red Hat Enterprise Linux
4 1 (0.36%) Windows
Processes & Methodologies
1 149 (53.02%) Data Science
2 87 (30.96%) Machine Learning
3 82 (29.18%) Analytics
4 76 (27.05%) Data Modelling
5 63 (22.42%) Agile Software Development
6 62 (22.06%) Data Engineering
7 56 (19.93%) Mentoring
8 54 (19.22%) Coaching
8 54 (19.22%) Data Cleansing
9 45 (16.01%) SQL Optimisation
10 42 (14.95%) DevOps
11 29 (10.32%) TDD
12 22 (7.83%) Artificial Intelligence
13 21 (7.47%) BDD
14 19 (6.76%) Microservices
15 18 (6.41%) ETL
15 18 (6.41%) Problem-Solving
16 17 (6.05%) Computer Science
17 15 (5.34%) Software Engineering
18 14 (4.98%) Business Intelligence
Programming Languages
1 167 (59.43%) Python
2 110 (39.15%) SQL
3 70 (24.91%) Java
4 61 (21.71%) Scala
5 20 (7.12%) R
6 15 (5.34%) Java 8
7 7 (2.49%) Shell Script
8 6 (2.14%) Bash Shell
8 6 (2.14%) JavaScript
9 5 (1.78%) C#
9 5 (1.78%) T-SQL
9 5 (1.78%) U-SQL
10 4 (1.42%) Kotlin
11 3 (1.07%) Clojure
12 2 (0.71%) ES6
12 2 (0.71%) PowerShell
12 2 (0.71%) TypeScript
12 2 (0.71%) VBA
13 1 (0.36%) Go
Qualifications
1 11 (3.91%) Degree
2 6 (2.14%) Numerate Degree
2 6 (2.14%) PhD
3 2 (0.71%) Computer Science Degree
3 2 (0.71%) Master's Degree
4 1 (0.36%) Postgraduate
Quality Assurance & Compliance
1 46 (16.37%) Data Quality
2 1 (0.36%) GDPR
2 1 (0.36%) QA
System Software
1 37 (13.17%) Docker
2 23 (8.19%) HDFS
3 7 (2.49%) Apache Flume
4 1 (0.36%) Active Directory
Systems Management
1 46 (16.37%) Oozie
2 20 (7.12%) Hadoop YARN
3 19 (6.76%) Ansible
4 18 (6.41%) Kubernetes
5 7 (2.49%) Terraform
6 5 (1.78%) Puppet
7 4 (1.42%) Kibana
8 3 (1.07%) logstash
8 3 (1.07%) Marathon
8 3 (1.07%) Mesos
9 2 (0.71%) Docker Swarm
10 1 (0.36%) Grafana
10 1 (0.36%) Istio
10 1 (0.36%) Kubeflow
10 1 (0.36%) rkt
10 1 (0.36%) Tivoli
Vendors
1 79 (28.11%) Google
2 22 (7.83%) Splunk
3 14 (4.98%) Oracle
4 9 (3.20%) Microsoft
5 6 (2.14%) Cloudera
5 6 (2.14%) Tableau
5 6 (2.14%) Talend
6 4 (1.42%) StreamSets
7 3 (1.07%) Atlassian
7 3 (1.07%) Informatica
7 3 (1.07%) Sybase
7 3 (1.07%) Teradata
8 2 (0.71%) Alteryx
8 2 (0.71%) Datameer
8 2 (0.71%) SAP
9 1 (0.36%) Databricks
9 1 (0.36%) Hortonworks
9 1 (0.36%) MapR