Period
to 7 December 2019

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 7 December 2019 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
England > London
6 months to
7 Dec 2019
Same period 2018 Same period 2017
Rank 135 141 124
Rank change year-on-year +6 -17 +14
Contract jobs citing Apache Spark 687 935 1,030
As % of all contract IT jobs advertised in London 2.01% 2.00% 2.17%
As % of the Application Platforms category 13.20% 11.60% 12.70%
Number of daily rates quoted 530 678 801
London median daily rate £575 £575 £550
Median daily rate % change year-on-year - +4.55% -
10th Percentile £438 £463 £425
90th Percentile £738 £725 £700
England median daily rate £575 £550 £550
% change year-on-year +4.55% - -
Number of hourly rates quoted 1 4 0
London median hourly rate £52.50 £50.00 -
Median hourly rate % change year-on-year +5.00% - -
England median hourly rate £29.83 £50.00 £23.42
% change year-on-year -40.35% +113.49% -61.13%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for application platform skills.

All Application Platform Skills
London
Contract vacancies with a requirement for application platform skills 5,205 8,058 8,110
As % of all contract IT jobs advertised in London 15.22% 17.22% 17.07%
Number of daily rates quoted 3,653 5,665 5,833
London median daily rate £500 £500 £500
10th Percentile £278 £303 £300
90th Percentile £675 £675 £644
England median daily rate £450 £475 £450
% change year-on-year -5.26% +5.56% +5.88%
Number of hourly rates quoted 61 92 106
London median hourly rate £25.00 £23.67 £21.46
Median hourly rate % change year-on-year +5.63% +10.31% +19.19%
10th Percentile £15.25 £13.08 £12.50
90th Percentile £50.00 £57.50 £47.83
England median hourly rate £20.50 £21.75 £20.00
% change year-on-year -5.75% +8.75% -

Apache Spark
Job Vacancy Trend in London

Job postings citing Apache Spark as a percentage of all IT jobs advertised in London.

Job vacancy trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Trend in London

This chart provides the 3-month moving average for daily rates quoted in contract jobs citing Apache Spark in London.

Contractor daily rate trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Histogram in London

The daily rate distribution of IT jobs citing Apache Spark in London over the 6 months to 7 December 2019.

Contractor daily rate histogram for Apache Spark in London

Apache Spark
Contractor Hourly Rate Trend in London

This chart provides the 3-month moving average for contractor hourly rates quoted in IT jobs citing Apache Spark in London.

Contractor hourly rate trend for Apache Spark in London

Apache Spark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the London region over the 6 months to 7 December 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -19 155 £575 - 69
East London +25 30 £575 +6.98% 16
West London +28 20 £550 - 27
South London +30 15 £525 -0.24% 3
Apache Spark
England

For the 6 months to 7 December 2019, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Apache Spark.

1 468 (68.12%) Python
2 421 (61.28%) Hadoop
3 413 (60.12%) Big Data
4 267 (38.86%) Java
5 265 (38.57%) AWS
6 256 (37.26%) Scala
6 256 (37.26%) SQL
7 254 (36.97%) Agile Software Development
8 252 (36.68%) Kafka
9 196 (28.53%) Data Science
10 186 (27.07%) ETL
11 182 (26.49%) Finance
12 168 (24.45%) Apache Hive
13 156 (22.71%) Machine Learning
14 150 (21.83%) Azure
15 147 (21.40%) Analytics
16 133 (19.36%) DevOps
17 132 (19.21%) GCP
18 131 (19.07%) Kubernetes
19 130 (18.92%) Docker
20 126 (18.34%) Apache
21 112 (16.30%) Computer Science
21 112 (16.30%) PySpark
22 111 (16.16%) Data Engineering
23 100 (14.56%) Google
24 99 (14.41%) Data Warehouse
25 98 (14.26%) NoSQL
26 93 (13.54%) Linux
27 83 (12.08%) Jenkins
28 82 (11.94%) Algorithms

Apache Spark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 126 (18.34%) Apache
2 59 (8.59%) Elasticsearch
3 39 (5.68%) Apache Airflow
4 21 (3.06%) Jupyter
5 14 (2.04%) Luigi
6 13 (1.89%) Confluence
7 11 (1.60%) SAS
8 8 (1.16%) Apache Solr
9 6 (0.87%) Apache Pig
9 6 (0.87%) Pachyderm
10 5 (0.73%) DNN
10 5 (0.73%) JBoss
10 5 (0.73%) Jetty
10 5 (0.73%) Oracle Self-Service HR
10 5 (0.73%) Tomcat
10 5 (0.73%) WebLogic
11 2 (0.29%) CMS
11 2 (0.29%) Tornado
12 1 (0.15%) Oracle Coherence
12 1 (0.15%) Oracle Workflow
Applications
1 15 (2.18%) Microsoft Excel
Business Applications
1 9 (1.31%) FINCAD Analytics
2 5 (0.73%) SAP Oil and Gas
3 3 (0.44%) Distributed Ledger
4 2 (0.29%) Adobe Campaign
5 1 (0.15%) Cognos Business Intelligence
5 1 (0.15%) NetSuite
Cloud Services
1 265 (38.57%) AWS
2 150 (21.83%) Azure
3 132 (19.21%) GCP
4 65 (9.46%) Amazon EMR
5 58 (8.44%) Amazon Kinesis
5 58 (8.44%) Amazon S3
5 58 (8.44%) Serverless
6 50 (7.28%) AWS Lambda
7 27 (3.93%) GitHub
8 25 (3.64%) Amazon EC2
9 24 (3.49%) OpenShift
10 22 (3.20%) AWS CloudFormation
11 19 (2.77%) Azure Data Factory
12 18 (2.62%) Power Platform
13 17 (2.47%) Amazon SQS
14 15 (2.18%) Amazon EKS
15 12 (1.75%) AWS Step Functions
16 11 (1.60%) Azure Machine Learning
17 10 (1.46%) CloudFront
18 9 (1.31%) Cloud Computing
Communications & Networking
1 6 (0.87%) HTTP
1 6 (0.87%) Kerberos
2 5 (0.73%) SSH
3 3 (0.44%) Firewall
4 2 (0.29%) F5 BIG-IP GTM
4 2 (0.29%) F5 BIG-IP LTM
4 2 (0.29%) Internet
4 2 (0.29%) Software-Defined Networking
5 1 (0.15%) SSL
5 1 (0.15%) VPN
Database & Business Intelligence
1 421 (61.28%) Hadoop
2 413 (60.12%) Big Data
3 168 (24.45%) Apache Hive
4 99 (14.41%) Data Warehouse
5 98 (14.26%) NoSQL
6 77 (11.21%) PostgreSQL
7 67 (9.75%) Amazon Redshift
7 67 (9.75%) MongoDB
8 66 (9.61%) Relational Database
9 63 (9.17%) Data Lake
10 60 (8.73%) Impala
11 56 (8.15%) HBase
12 51 (7.42%) Apache Cassandra
12 51 (7.42%) Flink
13 49 (7.13%) Spark SQL
14 47 (6.84%) Amazon Athena
15 41 (5.97%) SQL Server
16 38 (5.53%) BigQuery
17 34 (4.95%) Presto
18 29 (4.22%) MySQL
Development Applications
1 83 (12.08%) Jenkins
2 78 (11.35%) Git (software)
3 37 (5.39%) JIRA
4 25 (3.64%) Maven
5 16 (2.33%) Atlassian Bamboo
6 15 (2.18%) Vagrant
7 12 (1.75%) IntelliJ
8 9 (1.31%) Eclipse
8 9 (1.31%) Gradle
8 9 (1.31%) Mercurial
8 9 (1.31%) Selenium
9 8 (1.16%) Travis CI
10 7 (1.02%) Ant
11 6 (0.87%) FishEye
11 6 (0.87%) JUnit
11 6 (0.87%) Visual Studio Team System
12 5 (0.73%) ClearCase
12 5 (0.73%) Hudson
12 5 (0.73%) Poseidon for UML
12 5 (0.73%) Subversion
General
1 182 (26.49%) Finance
2 69 (10.04%) Banking
3 50 (7.28%) Front Office
4 27 (3.93%) Retail
5 23 (3.35%) Investment Banking
5 23 (3.35%) Legal
6 16 (2.33%) Telecoms
7 13 (1.89%) Retail Banking
8 8 (1.16%) Marketing
8 8 (1.16%) Pharmaceutical
9 7 (1.02%) Games
9 7 (1.02%) Publishing
10 6 (0.87%) Public Sector
11 4 (0.58%) Financial Institution
12 3 (0.44%) Advertising
13 2 (0.29%) Corporate Banking
13 2 (0.29%) Manufacturing
14 1 (0.15%) Automotive
14 1 (0.15%) Billing
14 1 (0.15%) International Banking
Job Titles
1 269 (39.16%) Data Engineer
2 190 (27.66%) Developer
3 82 (11.94%) Big Data Engineer
4 73 (10.63%) Architect
5 70 (10.19%) Python Developer
6 69 (10.04%) Python Engineer
7 57 (8.30%) Data Scientist
8 53 (7.71%) AWS Engineer
9 51 (7.42%) Java Developer
10 47 (6.84%) Data Architect
11 42 (6.11%) Big Data Developer
12 41 (5.97%) Senior Data Engineer
13 39 (5.68%) Senior Developer
14 30 (4.37%) Scala Developer
15 29 (4.22%) Hadoop Developer
16 25 (3.64%) Java Engineer
17 23 (3.35%) DevOps Engineer
18 22 (3.20%) Azure Engineer
18 22 (3.20%) Big Data Architect
19 21 (3.06%) Cloud Engineer
Libraries, Frameworks & Software Standards
1 252 (36.68%) Kafka
2 112 (16.30%) PySpark
3 74 (10.77%) Pandas
4 63 (9.17%) Spark Streaming
5 45 (6.55%) Spring
6 43 (6.26%) React
7 42 (6.11%) Web Services
8 39 (5.68%) Natural Language Processing
9 34 (4.95%) NumPy
10 31 (4.51%) TensorFlow
11 27 (3.93%) REST
11 27 (3.93%) RESTful
11 27 (3.93%) scikit-learn
12 26 (3.78%) AngularJS
13 25 (3.64%) Apache Beam
13 25 (3.64%) JSON
14 22 (3.20%) SciPy
15 19 (2.77%) Node.js
16 18 (2.62%) Flask
16 18 (2.62%) MLlib
Miscellaneous
1 82 (11.94%) Algorithms
2 59 (8.59%) Analytical Skills
3 48 (6.99%) Data Structures
4 41 (5.97%) Greenfield Project
5 21 (3.06%) Self-Motivation
6 19 (2.77%) User Experience
7 18 (2.62%) Derivative
8 15 (2.18%) Fixed Income
8 15 (2.18%) Hedge funds
8 15 (2.18%) Public Cloud
9 13 (1.89%) Foreign Exchange (FX)
10 11 (1.60%) Credit Risk
11 8 (1.16%) Grid Computing
11 8 (1.16%) Linux Command Line
11 8 (1.16%) Market Risk
12 6 (0.87%) Cloud Native
13 4 (0.58%) Blog
13 4 (0.58%) Reference Data
14 3 (0.44%) Equities
14 3 (0.44%) Equity Derivative
Operating Systems
1 93 (13.54%) Linux
2 51 (7.42%) Unix
3 30 (4.37%) Windows
4 11 (1.60%) Ubuntu
5 8 (1.16%) Red Hat Enterprise Linux
6 7 (1.02%) VMS
7 6 (0.87%) Android
7 6 (0.87%) Apple iOS
8 5 (0.73%) CentOS
8 5 (0.73%) Debian
8 5 (0.73%) Mac OS X
Processes & Methodologies
1 254 (36.97%) Agile Software Development
2 196 (28.53%) Data Science
3 186 (27.07%) ETL
4 156 (22.71%) Machine Learning
5 147 (21.40%) Analytics
6 133 (19.36%) DevOps
7 112 (16.30%) Computer Science
8 111 (16.16%) Data Engineering
9 80 (11.64%) TDD
10 72 (10.48%) Artificial Intelligence
10 72 (10.48%) Microservices
11 71 (10.33%) Data Modelling
12 68 (9.90%) Problem-Solving
13 63 (9.17%) Scrum
14 60 (8.73%) Open Source
15 58 (8.44%) Continuous Integration
15 58 (8.44%) Software Engineering
16 53 (7.71%) Migration
17 46 (6.70%) Visualisation
18 43 (6.26%) Business Intelligence
Programming Languages
1 468 (68.12%) Python
2 267 (38.86%) Java
3 256 (37.26%) Scala
3 256 (37.26%) SQL
4 65 (9.46%) R
5 49 (7.13%) JavaScript
6 31 (4.51%) C++
7 28 (4.08%) Go
8 27 (3.93%) C#
9 19 (2.77%) Bash Shell
10 18 (2.62%) Clojure
11 17 (2.47%) C
12 14 (2.04%) T-SQL
13 13 (1.89%) Ruby
14 11 (1.60%) Groovy
14 11 (1.60%) PL/SQL
15 10 (1.46%) Kotlin
16 9 (1.31%) Java 8
17 8 (1.16%) Perl
18 7 (1.02%) Shell Script
Qualifications
1 76 (11.06%) Degree
2 35 (5.09%) Computer Science Degree
3 32 (4.66%) PhD
4 26 (3.78%) Master's Degree
5 12 (1.75%) Security Cleared
6 10 (1.46%) AWS Certification
7 7 (1.02%) Numerate Degree
7 7 (1.02%) SC Cleared
8 5 (0.73%) PMI Certification
8 5 (0.73%) PMP
9 4 (0.58%) DV Cleared
10 2 (0.29%) CCIE
10 2 (0.29%) CCNP
10 2 (0.29%) Cisco Certification
10 2 (0.29%) DBS Check
11 1 (0.15%) CCA
11 1 (0.15%) Microsoft Certification
11 1 (0.15%) TOGAF Certification
Quality Assurance & Compliance
1 11 (1.60%) Data Quality
2 7 (1.02%) QA
3 5 (0.73%) MiFID
4 4 (0.58%) EMIR
5 1 (0.15%) GDPR
5 1 (0.15%) WCAG
System Software
1 130 (18.92%) Docker
2 67 (9.75%) HDFS
3 9 (1.31%) VirtualBox
4 7 (1.02%) Apache ZooKeeper
5 4 (0.58%) Apache Flume
6 2 (0.29%) VMware Infrastructure
Systems Management
1 131 (19.07%) Kubernetes
2 54 (7.86%) Ansible
3 49 (7.13%) Hadoop YARN
4 37 (5.39%) Terraform
5 18 (2.62%) Puppet
6 14 (2.04%) Oozie
7 10 (1.46%) Opscode Chef
8 9 (1.31%) Docker Swarm
9 6 (0.87%) Kubeflow
9 6 (0.87%) Salt
10 4 (0.58%) Cobbler
10 4 (0.58%) Red Hat Satellite
11 3 (0.44%) Graphite
11 3 (0.44%) Kibana
11 3 (0.44%) Mesos
11 3 (0.44%) Prometheus
12 2 (0.29%) FortiGate
12 2 (0.29%) Marathon
13 1 (0.15%) Ambari
13 1 (0.15%) ZABBIX
Vendors
1 100 (14.56%) Google
2 80 (11.64%) Oracle
3 63 (9.17%) Cloudera
4 55 (8.01%) Databricks
5 40 (5.82%) Hortonworks
6 35 (5.09%) Informatica
7 34 (4.95%) Microsoft
8 33 (4.80%) Tableau
9 28 (4.08%) Talend
10 22 (3.20%) IBM
11 21 (3.06%) Palantir
12 19 (2.77%) Red Hat
13 17 (2.47%) SAP
14 16 (2.33%) Teradata
15 15 (2.18%) Splunk
16 12 (1.75%) MapR
17 11 (1.60%) Alteryx
18 10 (1.46%) IBM Sterling
19 9 (1.31%) Bloomberg
19 9 (1.31%) Sybase