Period
to 23 September 2019

The following table provides summary statistics for contract job vacancies advertised in the City of London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 23 September 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
Central London > City of London
6 months to
23 Sep 2019
Same period 2018 Same period 2017
Rank 73 78 101
Rank change year-on-year +5 +23 +42
Contract jobs citing Apache Spark 208 216 161
As % of all contract IT jobs advertised in the City of London 3.35% 3.14% 2.39%
As % of the Application Platforms category 20.14% 18.27% 16.07%
Number of daily rates quoted 177 191 144
City of London median daily rate £560 £550 £600
Median daily rate % change year-on-year +1.82% -8.33% -
10th Percentile £513 £469 £472
90th Percentile £700 £725 £725
Central London median daily rate £566 £550 £600
% change year-on-year +2.95% -8.33% +2.13%
Number of hourly rates quoted 0 0 0
City of London median hourly rate - - -
Central London median hourly rate - - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in the City of London with a requirement for application platform skills.

All Application Platform Skills
City of London
Contract vacancies with a requirement for application platform skills 1,033 1,182 1,002
As % of all contract IT jobs advertised in the City of London 16.63% 17.21% 14.87%
Number of daily rates quoted 815 899 753
City of London median daily rate £525 £510 £500
Median daily rate % change year-on-year +2.94% +2.00% +11.11%
10th Percentile £300 £280 £268
90th Percentile £675 £675 £675
Central London median daily rate £525 £500 £498
% change year-on-year +5.00% +0.50% +10.56%
Number of hourly rates quoted 11 36 8
City of London median hourly rate £21.00 £20.63 £17.00
Median hourly rate % change year-on-year +1.82% +21.32% -6.85%
10th Percentile £12.98 £13.00 £12.01
90th Percentile £55.50 £33.38 £59.63
Central London median hourly rate £21.00 £20.00 £17.85
% change year-on-year +5.00% +12.04% +2.00%

Apache Spark
Job Vacancy Trend in the City of London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in the City of London.

Job vacancy trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Trend in the City of London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in the City of London.

Contractor daily rate trend for Apache Spark in the City of London

Apache Spark
Contractor Daily Rate Histogram in the City of London

The daily rate distribution of IT jobs citing Apache Spark in the City of London over the 6 months to 23 September 2019.

Contractor daily rate histogram for Apache Spark in the City of London

Apache Spark
Contract Job Locations in the City of London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the City of London region over the 6 months to 23 September 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
London Bridge +1 59 £550 -
Moorgate - 10 £575 -
Apache Spark
Central London

For the 6 months to 23 September 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the City of London region with a requirement for Apache Spark.

1 135 (64.90%) Big Data
2 127 (61.06%) Hadoop
3 117 (56.25%) Python
4 109 (52.40%) AWS
5 102 (49.04%) Data Science
6 95 (45.67%) Finance
7 93 (44.71%) SQL
8 90 (43.27%) Kafka
9 86 (41.35%) Apache Hive
10 74 (35.58%) Agile Software Development
11 70 (33.65%) Scala
12 69 (33.17%) Java
13 61 (29.33%) Analytics
14 56 (26.92%) Data Modelling
14 56 (26.92%) GCP
15 54 (25.96%) Machine Learning
16 50 (24.04%) Google
17 49 (23.56%) Azure
17 49 (23.56%) PySpark
18 42 (20.19%) Apache
19 41 (19.71%) Data Engineering
20 39 (18.75%) Docker
20 39 (18.75%) Data Warehouse
21 38 (18.27%) DevOps
22 35 (16.83%) Apache Airflow
23 34 (16.35%) Mentoring
24 29 (13.94%) Coaching
24 29 (13.94%) Kubernetes
24 29 (13.94%) Linux
24 29 (13.94%) Banking

Apache Spark
Co-occurring IT Skills in the City of London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 42 (20.19%) Apache
2 35 (16.83%) Apache Airflow
3 21 (10.10%) Elasticsearch
4 11 (5.29%) Confluence
5 4 (1.92%) Apache Solr
6 3 (1.44%) Jupyter
6 3 (1.44%) Luigi
7 2 (0.96%) Apache Pig
7 2 (0.96%) JBoss
7 2 (0.96%) Jetty
7 2 (0.96%) Oracle Self-Service HR
7 2 (0.96%) Tomcat
7 2 (0.96%) WebLogic
8 1 (0.48%) SAS
8 1 (0.48%) SharePoint
Applications
1 2 (0.96%) Microsoft Excel
Business Applications
1 1 (0.48%) Cognos Business Intelligence
1 1 (0.48%) SAP ERP
1 1 (0.48%) SAP Oil and Gas
1 1 (0.48%) SAP S/4HANA
Cloud Services
1 109 (52.40%) AWS
2 56 (26.92%) GCP
3 49 (23.56%) Azure
4 18 (8.65%) Amazon S3
5 15 (7.21%) Serverless
6 12 (5.77%) Amazon EMR
7 11 (5.29%) AWS Lambda
8 10 (4.81%) Amazon EC2
8 10 (4.81%) Amazon SageMaker
9 9 (4.33%) AWS CloudFormation
10 8 (3.85%) Amazon Kinesis
11 5 (2.40%) GitHub
12 4 (1.92%) Cloud Computing
12 4 (1.92%) Power Platform
13 3 (1.44%) Azure Data Factory
13 3 (1.44%) Datadog
13 3 (1.44%) PaaS
13 3 (1.44%) SaaS
14 2 (0.96%) Amazon CloudWatch
14 2 (0.96%) Google Analytics
Communications & Networking
1 2 (0.96%) SSH
2 1 (0.48%) DNS
2 1 (0.48%) Firewall
Database & Business Intelligence
1 135 (64.90%) Big Data
2 127 (61.06%) Hadoop
3 86 (41.35%) Apache Hive
4 39 (18.75%) Data Warehouse
5 27 (12.98%) NoSQL
5 27 (12.98%) OLAP
6 25 (12.02%) HBase
7 24 (11.54%) Enterprise Data Warehouse
7 24 (11.54%) OLAP Cube
7 24 (11.54%) Star Schema
8 21 (10.10%) PostgreSQL
9 15 (7.21%) Data Lake
9 15 (7.21%) Impala
9 15 (7.21%) Spark SQL
10 12 (5.77%) Amazon Redshift
10 12 (5.77%) MongoDB
10 12 (5.77%) MySQL
11 11 (5.29%) Apache Cassandra
12 10 (4.81%) Sqoop
13 8 (3.85%) SQL Server
Development Applications
1 27 (12.98%) Jenkins
2 22 (10.58%) JIRA
3 16 (7.69%) Git (software)
4 7 (3.37%) IBM UrbanCode
4 7 (3.37%) RStudio
4 7 (3.37%) Travis CI
5 6 (2.88%) Maven
6 5 (2.40%) Atlassian Bamboo
7 3 (1.44%) Cucumber
7 3 (1.44%) ERwin
7 3 (1.44%) Subversion
7 3 (1.44%) Vagrant
7 3 (1.44%) Visual Studio Team System
8 2 (0.96%) Ant
8 2 (0.96%) ClearCase
8 2 (0.96%) Eclipse
8 2 (0.96%) IntelliJ
8 2 (0.96%) Leiningen
8 2 (0.96%) Octopus Deploy
8 2 (0.96%) tSQLt
General
1 95 (45.67%) Finance
2 29 (13.94%) Banking
3 9 (4.33%) Retail
4 7 (3.37%) Investment Banking
4 7 (3.37%) Retail Banking
5 3 (1.44%) Front Office
5 3 (1.44%) Public Sector
6 2 (0.96%) Publishing
7 1 (0.48%) Back Office
7 1 (0.48%) Games
7 1 (0.48%) Marketing
Job Titles
1 112 (53.85%) Data Engineer
2 42 (20.19%) Python Engineer
3 41 (19.71%) Developer
4 37 (17.79%) Big Data Engineer
5 36 (17.31%) Hadoop Engineer
6 16 (7.69%) Java Developer
7 13 (6.25%) Architect
8 11 (5.29%) Big Data Developer
8 11 (5.29%) Java Engineer
9 10 (4.81%) AWS Engineer
9 10 (4.81%) Python Developer
9 10 (4.81%) Scala Developer
10 9 (4.33%) Hadoop Developer
10 9 (4.33%) Java Hadoop Developer
11 8 (3.85%) Data Architect
12 7 (3.37%) DevOps Engineer
12 7 (3.37%) Platform Engineer
12 7 (3.37%) Senior Developer
12 7 (3.37%) Senior Scala Developer
12 7 (3.37%) Unix Engineer
Libraries, Frameworks & Software Standards
1 90 (43.27%) Kafka
2 49 (23.56%) PySpark
3 28 (13.46%) Spring
4 17 (8.17%) Spark Streaming
5 16 (7.69%) scikit-learn
6 14 (6.73%) Play Framework
7 12 (5.77%) Pandas
7 12 (5.77%) Web Services
8 10 (4.81%) MLlib
9 9 (4.33%) Lambda Expressions
9 9 (4.33%) NumPy
10 8 (3.85%) React
10 8 (3.85%) Spring Boot
11 7 (3.37%) Java Concurrency
11 7 (3.37%) XML
12 6 (2.88%) Natural Language Processing
13 5 (2.40%) Elastic Stack
13 5 (2.40%) JSON
13 5 (2.40%) RESTful
14 4 (1.92%) Apache NiFi
Miscellaneous
1 24 (11.54%) Data Structures
2 20 (9.62%) Algorithms
3 13 (6.25%) Greenfield Project
4 9 (4.33%) Analytical Skills
5 8 (3.85%) Linux Command Line
6 6 (2.88%) Cloud Native
7 5 (2.40%) Foreign Exchange (FX)
7 5 (2.40%) Public Cloud
8 4 (1.92%) User Experience
9 2 (0.96%) Derivative
9 2 (0.96%) Enterprise Software
9 2 (0.96%) Self-Motivation
10 1 (0.48%) Market Risk
10 1 (0.48%) NHS
10 1 (0.48%) Social Media
10 1 (0.48%) Spread Betting
Operating Systems
1 29 (13.94%) Linux
2 28 (13.46%) Unix
3 3 (1.44%) Windows
4 2 (0.96%) Android
4 2 (0.96%) Apple iOS
4 2 (0.96%) CentOS
4 2 (0.96%) Debian
4 2 (0.96%) Mac OS X
4 2 (0.96%) Red Hat Enterprise Linux
4 2 (0.96%) Ubuntu
Processes & Methodologies
1 102 (49.04%) Data Science
2 74 (35.58%) Agile Software Development
3 61 (29.33%) Analytics
4 56 (26.92%) Data Modelling
5 54 (25.96%) Machine Learning
6 41 (19.71%) Data Engineering
7 38 (18.27%) DevOps
8 34 (16.35%) Mentoring
9 29 (13.94%) Coaching
10 28 (13.46%) Data Cleansing
11 26 (12.50%) TDD
12 25 (12.02%) ETL
13 24 (11.54%) SQL Optimisation
14 23 (11.06%) Microservices
15 17 (8.17%) Artificial Intelligence
16 15 (7.21%) Computer Science
17 13 (6.25%) BDD
18 12 (5.77%) Performance Tuning
18 12 (5.77%) Software Engineering
19 11 (5.29%) Fintech
Programming Languages
1 117 (56.25%) Python
2 93 (44.71%) SQL
3 70 (33.65%) Scala
4 69 (33.17%) Java
5 18 (8.65%) R
6 8 (3.85%) C++
6 8 (3.85%) Java 8
7 7 (3.37%) JavaScript
8 6 (2.88%) Clojure
8 6 (2.88%) Go
8 6 (2.88%) Shell Script
9 4 (1.92%) Bash Shell
9 4 (1.92%) Kotlin
9 4 (1.92%) T-SQL
10 3 (1.44%) C#
10 3 (1.44%) U-SQL
11 2 (0.96%) ClojureScript
11 2 (0.96%) Perl
11 2 (0.96%) PL/SQL
11 2 (0.96%) sed
Qualifications
1 8 (3.85%) Degree
2 3 (1.44%) Security Cleared
3 2 (0.96%) SC Cleared
4 1 (0.48%) CCA
4 1 (0.48%) DV Cleared
4 1 (0.48%) Master's Degree
4 1 (0.48%) Numerate Degree
4 1 (0.48%) PhD
Quality Assurance & Compliance
1 24 (11.54%) Data Quality
2 2 (0.96%) GDPR
3 1 (0.48%) QA
System Software
1 39 (18.75%) Docker
2 24 (11.54%) HDFS
3 3 (1.44%) Apache Flume
4 2 (0.96%) Apache ZooKeeper
5 1 (0.48%) Active Directory
5 1 (0.48%) VirtualBox
Systems Management
1 29 (13.94%) Kubernetes
2 27 (12.98%) Oozie
3 19 (9.13%) Hadoop YARN
4 17 (8.17%) Ansible
5 8 (3.85%) Terraform
6 4 (1.92%) Kibana
6 4 (1.92%) Puppet
7 3 (1.44%) logstash
7 3 (1.44%) Marathon
7 3 (1.44%) Mesos
8 2 (0.96%) Salt
9 1 (0.48%) Docker Swarm
9 1 (0.48%) Grafana
9 1 (0.48%) Istio
9 1 (0.48%) Kubeflow
9 1 (0.48%) Opscode Chef
9 1 (0.48%) Prometheus
Vendors
1 50 (24.04%) Google
2 16 (7.69%) Cloudera
3 13 (6.25%) Oracle
4 8 (3.85%) Tableau
5 6 (2.88%) Microsoft
6 5 (2.40%) Hortonworks
6 5 (2.40%) Informatica
6 5 (2.40%) Talend
7 4 (1.92%) Splunk
7 4 (1.92%) Teradata
8 2 (0.96%) Databricks
8 2 (0.96%) Red Hat
8 2 (0.96%) Sybase
9 1 (0.48%) Alteryx
9 1 (0.48%) Cognos
9 1 (0.48%) Datactics
9 1 (0.48%) IBM
9 1 (0.48%) New Relic
9 1 (0.48%) SAP
9 1 (0.48%) StreamSets