Period
to 21 September 2019

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 21 September 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
England > London
6 months to
21 Sep 2019
Same period 2018 Same period 2017
Rank 99 135 125
Rank change year-on-year +36 -10 +63
Contract jobs citing Apache Spark 980 966 966
As % of all contract IT jobs advertised in London 2.55% 2.11% 2.18%
As % of the Application Platforms category 16.46% 12.21% 12.63%
Number of daily rates quoted 768 717 744
London median daily rate £575 £550 £550
Median daily rate % change year-on-year +4.55% - -
10th Percentile £463 £450 £411
90th Percentile £713 £715 £701
England median daily rate £550 £550 £550
Number of hourly rates quoted 0 2 0
London median hourly rate - £30.25 -
England median hourly rate £24.65 £22.50 £19.34
% change year-on-year +9.56% +16.34% -5.66%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for application platform skills.

All Application Platform Skills
London
Contract vacancies with a requirement for application platform skills 5,953 7,910 7,649
As % of all contract IT jobs advertised in London 15.47% 17.25% 17.24%
Number of daily rates quoted 4,132 5,595 5,397
London median daily rate £500 £500 £488
Median daily rate % change year-on-year - +2.56% +8.33%
10th Percentile £300 £300 £300
90th Percentile £675 £663 £638
England median daily rate £463 £460 £450
% change year-on-year +0.54% +2.22% +5.88%
Number of hourly rates quoted 53 107 100
London median hourly rate £21.50 £22.50 £20.47
Median hourly rate % change year-on-year -4.44% +9.93% +13.71%
10th Percentile £15.51 £13.00 £12.73
90th Percentile £52.00 £63.50 £47.83
England median hourly rate £20.68 £21.50 £20.47
% change year-on-year -3.84% +5.04% +2.34%

Apache Spark
Job Vacancy Trend in London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in London.

Job vacancy trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Trend in London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in London.

Contractor daily rate trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Histogram in London

The daily rate distribution of IT jobs citing Apache Spark in London over the 6 months to 21 September 2019.

Contractor daily rate histogram for Apache Spark in London

Apache Spark
Contractor Hourly Rate Trend in London

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark in London.

Contractor hourly rate trend for Apache Spark in London

Apache Spark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the London region over the 6 months to 21 September 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London +19 268 £570 +3.64% 72
West London +34 40 £525 -4.55% 19
East London -13 24 £575 +4.55% 11
South London +32 15 £525 -0.94% 3
North London -1 1 - - 1
Apache Spark
England

For the 6 months to 21 September 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Apache Spark.

1 639 (65.20%) Big Data
2 631 (64.39%) Python
3 592 (60.41%) Hadoop
4 396 (40.41%) SQL
5 387 (39.49%) AWS
6 371 (37.86%) Kafka
7 361 (36.84%) Agile Software Development
8 344 (35.10%) Java
9 327 (33.37%) Scala
10 312 (31.84%) Data Science
11 296 (30.20%) Apache Hive
12 281 (28.67%) Finance
13 242 (24.69%) Analytics
14 233 (23.78%) GCP
15 228 (23.27%) Azure
16 221 (22.55%) Machine Learning
17 220 (22.45%) DevOps
18 204 (20.82%) ETL
19 188 (19.18%) Apache
19 188 (19.18%) Docker
20 181 (18.47%) Data Engineering
21 175 (17.86%) Data Warehouse
22 172 (17.55%) Data Modelling
23 171 (17.45%) Google
24 154 (15.71%) Kubernetes
25 150 (15.31%) NoSQL
26 141 (14.39%) Linux
27 137 (13.98%) PySpark
28 132 (13.47%) Computer Science
29 112 (11.43%) Git (software)

Apache Spark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 188 (19.18%) Apache
2 75 (7.65%) Apache Airflow
3 73 (7.45%) Elasticsearch
4 24 (2.45%) SAS
5 23 (2.35%) Confluence
6 22 (2.24%) Apache Solr
7 15 (1.53%) Jupyter
8 13 (1.33%) Luigi
9 12 (1.22%) Apache Pig
10 9 (0.92%) Oracle Coherence
10 9 (0.92%) Pachyderm
11 4 (0.41%) JBoss
11 4 (0.41%) Jetty
11 4 (0.41%) Oracle Self-Service HR
11 4 (0.41%) SharePoint
11 4 (0.41%) Tomcat
11 4 (0.41%) WebLogic
12 2 (0.20%) ArcGIS
12 2 (0.20%) OpenStack
13 1 (0.10%) Oracle Workflow
Applications
1 22 (2.24%) Microsoft Excel
Business Applications
1 6 (0.61%) SAP Oil and Gas
2 3 (0.31%) FINCAD Analytics
3 2 (0.20%) Adobe Campaign
4 1 (0.10%) Cognos Business Intelligence
4 1 (0.10%) Oracle Retail
4 1 (0.10%) SAP ERP
4 1 (0.10%) SAP S/4HANA
Cloud Services
1 387 (39.49%) AWS
2 233 (23.78%) GCP
3 228 (23.27%) Azure
4 105 (10.71%) Serverless
5 82 (8.37%) Amazon S3
6 81 (8.27%) Amazon Kinesis
7 80 (8.16%) AWS Lambda
8 71 (7.24%) Amazon EMR
9 36 (3.67%) Azure Data Factory
10 30 (3.06%) Amazon EC2
11 28 (2.86%) CloudFront
12 27 (2.76%) GitHub
13 26 (2.65%) Power Platform
14 19 (1.94%) Cloud Computing
15 17 (1.73%) AWS CloudFormation
16 15 (1.53%) Amazon SageMaker
17 13 (1.33%) OpenShift
17 13 (1.33%) SaaS
18 12 (1.22%) Azure Functions
19 11 (1.12%) Amazon EKS
Communications & Networking
1 8 (0.82%) HTTP
1 8 (0.82%) TCP/IP
2 7 (0.71%) Kerberos
3 5 (0.51%) Internet
4 4 (0.41%) Firewall
4 4 (0.41%) SSH
5 3 (0.31%) Software-Defined Networking
6 2 (0.20%) F5 BIG-IP GTM
6 2 (0.20%) F5 BIG-IP LTM
7 1 (0.10%) DNS
7 1 (0.10%) LAN
7 1 (0.10%) SSL
7 1 (0.10%) VPN
7 1 (0.10%) WAN
Database & Business Intelligence
1 639 (65.20%) Big Data
2 592 (60.41%) Hadoop
3 296 (30.20%) Apache Hive
4 175 (17.86%) Data Warehouse
5 150 (15.31%) NoSQL
6 110 (11.22%) Data Lake
7 98 (10.00%) PostgreSQL
8 96 (9.80%) Impala
9 82 (8.37%) Apache Cassandra
9 82 (8.37%) HBase
10 80 (8.16%) MongoDB
11 76 (7.76%) Amazon Redshift
12 67 (6.84%) Relational Database
13 61 (6.22%) BigQuery
14 54 (5.51%) SQL Server
15 43 (4.39%) Spark SQL
16 40 (4.08%) Flink
17 35 (3.57%) MySQL
18 34 (3.47%) Amazon Athena
19 32 (3.27%) Sqoop
Development Applications
1 112 (11.43%) Git (software)
2 108 (11.02%) Jenkins
3 50 (5.10%) JIRA
4 30 (3.06%) Atlassian Bamboo
5 17 (1.73%) Maven
5 17 (1.73%) RStudio
6 15 (1.53%) Vagrant
7 14 (1.43%) GitLab
8 13 (1.33%) TeamCity
9 12 (1.22%) Travis CI
10 11 (1.12%) Visual Studio Team System
11 10 (1.02%) IBM UrbanCode
12 9 (0.92%) IntelliJ
13 8 (0.82%) CircleCI
13 8 (0.82%) Eclipse
14 7 (0.71%) ERwin
14 7 (0.71%) Subversion
15 6 (0.61%) Ant
15 6 (0.61%) JUnit
15 6 (0.61%) Selenium
General
1 281 (28.67%) Finance
2 101 (10.31%) Banking
3 50 (5.10%) Front Office
4 36 (3.67%) Investment Banking
4 36 (3.67%) Retail
5 29 (2.96%) Legal
6 14 (1.43%) Financial Institution
7 13 (1.33%) Retail Banking
8 12 (1.22%) Marketing
8 12 (1.22%) Telecoms
9 11 (1.12%) Advertising
10 10 (1.02%) Publishing
11 8 (0.82%) Public Sector
12 7 (0.71%) Games
13 4 (0.41%) Aerospace
13 4 (0.41%) Manufacturing
14 3 (0.31%) Pharmaceutical
15 2 (0.20%) Automotive
15 2 (0.20%) Back Office
16 1 (0.10%) Social Science
Job Titles
1 431 (43.98%) Data Engineer
2 235 (23.98%) Developer
3 151 (15.41%) Big Data Engineer
4 122 (12.45%) Python Engineer
5 85 (8.67%) Architect
6 83 (8.47%) AWS Engineer
7 74 (7.55%) Python Developer
8 69 (7.04%) Hadoop Engineer
9 67 (6.84%) Java Developer
10 62 (6.33%) Big Data Developer
11 55 (5.61%) Data Scientist
12 53 (5.41%) Data Architect
12 53 (5.41%) Senior Developer
13 51 (5.20%) DevOps Engineer
14 46 (4.69%) Senior Data Engineer
15 45 (4.59%) Cloud Engineer
16 32 (3.27%) Hadoop Developer
17 31 (3.16%) Platform Engineer
18 29 (2.96%) AWS DevOps
19 28 (2.86%) Azure Engineer
Libraries, Frameworks & Software Standards
1 371 (37.86%) Kafka
2 137 (13.98%) PySpark
3 70 (7.14%) Spark Streaming
4 59 (6.02%) Pandas
5 58 (5.92%) Spring
6 53 (5.41%) Natural Language Processing
7 45 (4.59%) Web Services
8 36 (3.67%) React
8 36 (3.67%) scikit-learn
9 34 (3.47%) Spring Boot
9 34 (3.47%) TensorFlow
10 32 (3.27%) RESTful
11 28 (2.86%) NumPy
12 27 (2.76%) JSON
12 27 (2.76%) REST
13 23 (2.35%) Apache NiFi
13 23 (2.35%) Play Framework
14 22 (2.24%) HTML
14 22 (2.24%) MLlib
14 22 (2.24%) Node.js
Miscellaneous
1 109 (11.12%) Algorithms
2 64 (6.53%) Analytical Skills
3 63 (6.43%) Data Structures
4 61 (6.22%) Greenfield Project
5 27 (2.76%) Self-Motivation
6 24 (2.45%) Public Cloud
7 19 (1.94%) User Experience
8 18 (1.84%) Grid Computing
9 16 (1.63%) Foreign Exchange (FX)
9 16 (1.63%) Linux Command Line
10 15 (1.53%) Derivative
10 15 (1.53%) Market Risk
11 13 (1.33%) Cloud Native
12 12 (1.22%) Hedge funds
13 8 (0.82%) Credit Risk
13 8 (0.82%) Enterprise Software
13 8 (0.82%) Fixed Income
14 6 (0.61%) Virtual Team
15 5 (0.51%) Equities
15 5 (0.51%) Equity Derivative
Operating Systems
1 141 (14.39%) Linux
2 79 (8.06%) Unix
3 50 (5.10%) Windows
4 12 (1.22%) Red Hat Enterprise Linux
5 8 (0.82%) Ubuntu
6 7 (0.71%) VMS
7 5 (0.51%) Android
8 4 (0.41%) Apple iOS
8 4 (0.41%) CentOS
8 4 (0.41%) Debian
8 4 (0.41%) Mac OS X
9 1 (0.10%) Mesosphere
9 1 (0.10%) Windows Server
Processes & Methodologies
1 361 (36.84%) Agile Software Development
2 312 (31.84%) Data Science
3 242 (24.69%) Analytics
4 221 (22.55%) Machine Learning
5 220 (22.45%) DevOps
6 204 (20.82%) ETL
7 181 (18.47%) Data Engineering
8 172 (17.55%) Data Modelling
9 132 (13.47%) Computer Science
10 107 (10.92%) TDD
11 97 (9.90%) Artificial Intelligence
12 95 (9.69%) Scrum
13 88 (8.98%) Problem-Solving
14 80 (8.16%) Microservices
15 78 (7.96%) Software Engineering
16 65 (6.63%) Migration
17 59 (6.02%) Business Intelligence
18 56 (5.71%) Open Source
19 55 (5.61%) OO
20 53 (5.41%) Continuous Integration
Programming Languages
1 631 (64.39%) Python
2 396 (40.41%) SQL
3 344 (35.10%) Java
4 327 (33.37%) Scala
5 96 (9.80%) R
6 64 (6.53%) JavaScript
7 37 (3.78%) C++
8 36 (3.67%) C#
9 32 (3.27%) Bash Shell
10 31 (3.16%) Go
11 21 (2.14%) Java 8
12 20 (2.04%) Clojure
12 20 (2.04%) T-SQL
13 16 (1.63%) C
13 16 (1.63%) Kotlin
14 15 (1.53%) Ruby
15 14 (1.43%) MATLAB
16 13 (1.33%) PL/SQL
16 13 (1.33%) Shell Script
17 11 (1.12%) Julia
Qualifications
1 83 (8.47%) Degree
2 39 (3.98%) Computer Science Degree
3 36 (3.67%) PhD
4 28 (2.86%) AWS Certification
5 27 (2.76%) Security Cleared
6 22 (2.24%) Master's Degree
6 22 (2.24%) SC Cleared
7 18 (1.84%) PMI Certification
7 18 (1.84%) PMP
8 10 (1.02%) TOGAF Certification
9 3 (0.31%) DV Cleared
9 3 (0.31%) Numerate Degree
10 2 (0.20%) CCIE
10 2 (0.20%) CCNP
10 2 (0.20%) Cisco Certification
10 2 (0.20%) DBS Check
11 1 (0.10%) CCA
11 1 (0.10%) Microsoft Certification
Quality Assurance & Compliance
1 36 (3.67%) Data Quality
2 10 (1.02%) GDPR
3 4 (0.41%) MiFID
4 3 (0.31%) EMIR
4 3 (0.31%) QA
System Software
1 188 (19.18%) Docker
2 92 (9.39%) HDFS
3 34 (3.47%) Apache Flume
4 8 (0.82%) VirtualBox
5 7 (0.71%) Apache ZooKeeper
6 4 (0.41%) Active Directory
7 2 (0.20%) VMware Infrastructure
Systems Management
1 154 (15.71%) Kubernetes
2 69 (7.04%) Ansible
3 61 (6.22%) Hadoop YARN
4 52 (5.31%) Terraform
5 50 (5.10%) Oozie
6 21 (2.14%) Puppet
7 13 (1.33%) Kibana
8 10 (1.02%) Kubeflow
9 9 (0.92%) Opscode Chef
10 8 (0.82%) Docker Swarm
11 7 (0.71%) logstash
12 6 (0.61%) Marathon
12 6 (0.61%) Mesos
13 5 (0.51%) Grafana
14 4 (0.41%) Ambari
14 4 (0.41%) Salt
15 3 (0.31%) Cobbler
15 3 (0.31%) Prometheus
15 3 (0.31%) Red Hat Satellite
16 2 (0.20%) Istio
Vendors
1 171 (17.45%) Google
2 111 (11.33%) Cloudera
3 93 (9.49%) Oracle
4 74 (7.55%) Databricks
5 73 (7.45%) Informatica
6 51 (5.20%) Talend
7 43 (4.39%) Tableau
8 42 (4.29%) SAP
9 40 (4.08%) Hortonworks
10 38 (3.88%) IBM
10 38 (3.88%) Microsoft
11 28 (2.86%) IBM Sterling
12 18 (1.84%) MapR
12 18 (1.84%) Splunk
13 17 (1.73%) Red Hat
14 14 (1.43%) Sybase
14 14 (1.43%) Teradata
15 11 (1.12%) Alteryx
16 10 (1.02%) Atlassian
16 10 (1.02%) StreamSets