Period
to 1 August 2021

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 1 August 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
England > London
6 months to
1 Aug 2021
Same period 2020 Same period 2019
Rank 130 130 102
Rank change year-on-year 0 -28 +38
Contract jobs citing Apache Spark 630 334 1,046
As % of all contract jobs advertised in London 2.09% 2.12% 2.60%
As % of the Application Platforms category 15.22% 14.10% 16.43%
Number of daily rates quoted 397 219 796
10th Percentile £459 £420 £465
25th Percentile £513 £478 £513
Median daily rate (50th Percentile) £598 £550 £575
Median % change year-on-year +8.64% -4.35% +2.22%
75th Percentile £675 £659 £650
90th Percentile £725 £703 £710
England median daily rate £575 £533 £550
% change year-on-year +7.98% -3.18% -
Number of hourly rates quoted 2 0 0
Median hourly rate £37.50 - -
England median hourly rate £28.25 £10.50 £23.33
% change year-on-year +169.05% -54.98% +3.67%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for application platform skills.

All Application Platform Skills
London
Contract vacancies with a requirement for application platform skills 4,138 2,369 6,367
As % of all contract IT jobs advertised in London 13.74% 15.04% 15.83%
Number of daily rates quoted 2,899 1,635 4,439
10th Percentile £335 £321 £305
25th Percentile £430 £420 £413
Median daily rate (50th Percentile) £525 £500 £513
Median % change year-on-year +5.00% -2.44% +2.50%
75th Percentile £625 £600 £588
90th Percentile £700 £700 £675
England median daily rate £500 £475 £475
% change year-on-year +5.26% - +5.56%
Number of hourly rates quoted 31 21 47
10th Percentile £16.28 £15.00 £12.89
25th Percentile £21.13 £23.50 £17.00
Median hourly rate £32.00 £34.90 £20.89
Median % change year-on-year -8.30% +67.04% +1.90%
75th Percentile £57.50 £51.75 £34.80
90th Percentile £69.54 £60.00 £48.70
England median hourly rate £22.50 £20.00 £19.50
% change year-on-year +12.50% +2.56% -2.50%

Apache Spark
Job Vacancy Trend in London

Job postings citing Apache Spark as a proportion of all IT jobs advertised in London.

Job vacancy trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Trend in London

3-month moving average daily rate quoted in jobs citing Apache Spark in London.

Daily rate trend for Apache Spark in London

Apache Spark
Daily Rate Histogram in London

Daily rate distribution for jobs citing Apache Spark in London over the 6 months to 1 August 2021.

Daily rate histogram for Apache Spark in London

Apache Spark
Contractor Hourly Rate Trend in London

3-month moving average hourly rates quoted in jobs citing Apache Spark in London.

Hourly rate trend for Apache Spark in London

Apache Spark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the London region over the 6 months to 1 August 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -22 68 £600 +9.09% 29
West London 0 7 £640 +28.00% 9
South London +26 6 £625 +25.00% 1
East London 0 1 - - 5
North London - 1 - -
Apache Spark
England

For the 6 months to 1 August 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Apache Spark.

1 335 (53.17%) SQL
2 331 (52.54%) Big Data
3 326 (51.75%) Python
4 277 (43.97%) Hadoop
5 261 (41.43%) Kafka
6 245 (38.89%) Azure
7 240 (38.10%) Analytics
8 239 (37.94%) Scala
9 232 (36.83%) Java
10 203 (32.22%) Agile Software Development
11 187 (29.68%) Finance
12 186 (29.52%) Apache Hive
13 160 (25.40%) Data Pipeline
14 156 (24.76%) AWS
15 152 (24.13%) Apache
16 150 (23.81%) Social Skills
16 150 (23.81%) DevOps
17 139 (22.06%) Kubernetes
18 127 (20.16%) Data Science
19 125 (19.84%) Databricks
20 124 (19.68%) Banking
21 123 (19.52%) NoSQL
22 121 (19.21%) Data Warehouse
23 118 (18.73%) GCP
24 113 (17.94%) Data Modelling
25 112 (17.78%) Data Engineering
26 108 (17.14%) Docker
27 107 (16.98%) Apache Airflow
28 106 (16.83%) Jenkins
29 101 (16.03%) Git (software)

Apache Spark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 152 (24.13%) Apache
2 107 (16.98%) Apache Airflow
3 26 (4.13%) Elasticsearch
4 25 (3.97%) Luigi
5 23 (3.65%) Confluence
6 18 (2.86%) Oracle Coherence
7 15 (2.38%) Apache Pig
8 12 (1.90%) SharePoint
9 10 (1.59%) SAS
9 10 (1.59%) Tomcat
10 9 (1.43%) GemFire
11 5 (0.79%) Mule
11 5 (0.79%) Oracle Workflow
12 4 (0.63%) Jupyter
13 3 (0.48%) Apache Solr
13 3 (0.48%) JBoss
14 2 (0.32%) Cloud Foundry
15 1 (0.16%) DataSynapse
Applications
1 3 (0.48%) SPSS
2 2 (0.32%) Microsoft PowerPoint
3 1 (0.16%) SPSS Modeler
Business Applications
1 8 (1.27%) SAP Oil and Gas
2 2 (0.32%) SAP S/4HANA
3 1 (0.16%) Distributed Ledger
3 1 (0.16%) Oracle Cash Management
3 1 (0.16%) SAP R/3
3 1 (0.16%) Towers Watson Igloo
Cloud Services
1 245 (38.89%) Azure
2 156 (24.76%) AWS
3 118 (18.73%) GCP
4 80 (12.70%) Serverless
5 64 (10.16%) Azure Functions
6 60 (9.52%) Azure Cosmos DB
7 51 (8.10%) Azure Data Factory
8 48 (7.62%) Amazon S3
9 31 (4.92%) Amazon EMR
9 31 (4.92%) GitHub
10 24 (3.81%) Azure Synapse Analytics
10 24 (3.81%) Power Platform
11 23 (3.65%) Amazon EC2
12 22 (3.49%) Azure Storage
13 21 (3.33%) AWS Lambda
13 21 (3.33%) Snowflake
14 19 (3.02%) Collibra
15 11 (1.75%) Amazon Kinesis
15 11 (1.75%) Azure Machine Learning
15 11 (1.75%) Cloud Computing
Communications & Networking
1 10 (1.59%) Firewall
2 9 (1.43%) HTTP
3 5 (0.79%) SSL
4 4 (0.63%) FTP
5 3 (0.48%) DNS
5 3 (0.48%) HTTPS
5 3 (0.48%) Kerberos
6 2 (0.32%) Internet
6 2 (0.32%) Intranet
6 2 (0.32%) Network Security
7 1 (0.16%) Cisco ASA
7 1 (0.16%) IPv4
7 1 (0.16%) IPv6
7 1 (0.16%) SIGTRAN
7 1 (0.16%) SMPP
7 1 (0.16%) SNMP
7 1 (0.16%) SS7
7 1 (0.16%) SSH
7 1 (0.16%) TCP/IP
7 1 (0.16%) Wireshark
Database & Business Intelligence
1 331 (52.54%) Big Data
2 277 (43.97%) Hadoop
3 186 (29.52%) Apache Hive
4 123 (19.52%) NoSQL
5 121 (19.21%) Data Warehouse
6 78 (12.38%) Data Lake
7 76 (12.06%) Relational Database
8 65 (10.32%) PostgreSQL
9 63 (10.00%) MongoDB
10 62 (9.84%) HBase
11 60 (9.52%) BigQuery
12 51 (8.10%) Apache Cassandra
13 49 (7.78%) SQL Server
14 43 (6.83%) Spark SQL
15 42 (6.67%) Metadata
16 39 (6.19%) Parquet
16 39 (6.19%) Tableau
17 33 (5.24%) Amazon Redshift
17 33 (5.24%) Impala
18 30 (4.76%) MySQL
Development Applications
1 106 (16.83%) Jenkins
2 101 (16.03%) Git (software)
3 40 (6.35%) JIRA
4 27 (4.29%) Maven
5 24 (3.81%) TeamCity
6 23 (3.65%) Hudson
7 21 (3.33%) PowerDesigner
8 18 (2.86%) Subversion
9 15 (2.38%) ERwin
10 13 (2.06%) Artifactory
10 13 (2.06%) IntelliJ
10 13 (2.06%) Visual Studio Team System
11 12 (1.90%) Bitbucket
11 12 (1.90%) GitLab
12 11 (1.75%) Sonatype Nexus
13 8 (1.27%) MLflow
13 8 (1.27%) Oracle APEX
14 7 (1.11%) IBM UrbanCode
14 7 (1.11%) Mockito
15 6 (0.95%) ProC
General
1 187 (29.68%) Finance
2 150 (23.81%) Social Skills
3 124 (19.68%) Banking
4 96 (15.24%) Analytical Skills
5 36 (5.71%) Front Office
5 36 (5.71%) Investment Banking
6 31 (4.92%) Presentation Skills
7 22 (3.49%) Marketing
8 16 (2.54%) Public Sector
9 14 (2.22%) Financial Institution
10 11 (1.75%) Retail Banking
11 7 (1.11%) Games
12 6 (0.95%) Back Office
13 5 (0.79%) French Language
13 5 (0.79%) Publishing
14 4 (0.63%) Legal
15 3 (0.48%) Electronics
15 3 (0.48%) Influencing Skills
15 3 (0.48%) Local Government
15 3 (0.48%) Retail
Job Titles
1 228 (36.19%) Data Engineer
2 157 (24.92%) Developer
3 70 (11.11%) Architect
4 59 (9.37%) Java Developer
5 57 (9.05%) Big Data Engineer
6 48 (7.62%) Scala Developer
7 46 (7.30%) Data Architect
8 42 (6.67%) Data Scientist
9 39 (6.19%) Azure Engineer
10 29 (4.60%) Apache Spark Developer
11 27 (4.29%) Senior Developer
12 25 (3.97%) DevOps Engineer
13 23 (3.65%) Big Data Developer
14 21 (3.33%) Azure Architect
15 20 (3.17%) AWS Engineer
16 19 (3.02%) Data Modeller
17 18 (2.86%) Hadoop Engineer
17 18 (2.86%) Python Developer
17 18 (2.86%) Senior Data Engineer
18 17 (2.70%) Cloud Engineer
Libraries, Frameworks & Software Standards
1 261 (41.43%) Kafka
2 73 (11.59%) PySpark
3 55 (8.73%) Spring
4 47 (7.46%) Spark Streaming
5 29 (4.60%) JSON
5 29 (4.60%) RESTful
5 29 (4.60%) Spring Boot
6 25 (3.97%) Apache Avro
7 20 (3.17%) Akka
8 18 (2.86%) .NET
9 17 (2.70%) Pandas
9 17 (2.70%) Velocity
10 16 (2.54%) Web Services
11 14 (2.22%) Node.js
11 14 (2.22%) NumPy
12 13 (2.06%) OAuth
13 12 (1.90%) TensorFlow
14 11 (1.75%) Elastic Stack
14 11 (1.75%) OAuth2
14 11 (1.75%) YAML
Miscellaneous
1 57 (9.05%) Algorithms
2 53 (8.41%) Data Structures
3 41 (6.51%) Virtual Team
4 31 (4.92%) Market Risk
5 20 (3.17%) Management Information System
6 19 (3.02%) Grid Computing
7 15 (2.38%) Legacy Applications
8 14 (2.22%) Cloud Native
8 14 (2.22%) Legacy Systems
9 13 (2.06%) Data Centre
9 13 (2.06%) Greenfield Project
9 13 (2.06%) Self-Motivation
10 10 (1.59%) Hedge funds
10 10 (1.59%) Replication
11 9 (1.43%) Equities
11 9 (1.43%) Foreign Exchange (FX)
12 8 (1.27%) Fixed Income
12 8 (1.27%) Linux Command Line
12 8 (1.27%) Public Cloud
13 7 (1.11%) Data Protection Act
Operating Systems
1 59 (9.37%) Linux
2 43 (6.83%) Unix
3 12 (1.90%) Windows
4 6 (0.95%) Red Hat Enterprise Linux
5 3 (0.48%) Debian
5 3 (0.48%) Ubuntu
5 3 (0.48%) VMS
6 1 (0.16%) Android
Processes & Methodologies
1 240 (38.10%) Analytics
2 203 (32.22%) Agile Software Development
3 160 (25.40%) Data Pipeline
4 150 (23.81%) DevOps
5 127 (20.16%) Data Science
6 113 (17.94%) Data Modelling
7 112 (17.78%) Data Engineering
8 100 (15.87%) ETL
8 100 (15.87%) Machine Learning
9 84 (13.33%) Business Intelligence
10 80 (12.70%) Computer Science
10 80 (12.70%) Scrum
11 76 (12.06%) Data Architecture
12 75 (11.90%) Microservices
13 63 (10.00%) Open Source
14 60 (9.52%) TDD
15 57 (9.05%) Software Engineering
15 57 (9.05%) Visualisation
16 56 (8.89%) Artificial Intelligence
17 55 (8.73%) Data Management
Programming Languages
1 335 (53.17%) SQL
2 326 (51.75%) Python
3 239 (37.94%) Scala
4 232 (36.83%) Java
5 43 (6.83%) R
6 30 (4.76%) C#
7 25 (3.97%) Java 8
8 24 (3.81%) T-SQL
9 23 (3.65%) PL/SQL
10 21 (3.33%) C++
11 19 (3.02%) JavaScript
12 18 (2.86%) Go
13 15 (2.38%) Kotlin
14 12 (1.90%) Perl
15 11 (1.75%) Bash Shell
15 11 (1.75%) Shell Script
16 9 (1.43%) CQL
16 9 (1.43%) Groovy
17 8 (1.27%) C
18 7 (1.11%) Ruby
Qualifications
1 73 (11.59%) Degree
2 33 (5.24%) Computer Science Degree
3 18 (2.86%) Security Cleared
4 14 (2.22%) Numerate Degree
5 11 (1.75%) SC Cleared
6 10 (1.59%) ISEB
7 9 (1.43%) PhD
8 6 (0.95%) Master's Degree
9 3 (0.48%) DBS Check
10 2 (0.32%) AWS Certification
11 1 (0.16%) BPSS Clearance
11 1 (0.16%) CBAP
11 1 (0.16%) CCA
11 1 (0.16%) PMI Certification
Quality Assurance & Compliance
1 40 (6.35%) Data Quality
2 28 (4.44%) GDPR
3 22 (3.49%) QA
4 8 (1.27%) IFRS
5 3 (0.48%) MiFID
6 2 (0.32%) COBIT
6 2 (0.32%) ISO/IEC 27001
6 2 (0.32%) NIST
6 2 (0.32%) SLA
7 1 (0.16%) Dodd-Frank
7 1 (0.16%) EMIR
7 1 (0.16%) ISO 20022
7 1 (0.16%) PSD2
7 1 (0.16%) TARGET2
System Software
1 108 (17.14%) Docker
2 61 (9.68%) HDFS
3 5 (0.79%) Active Directory
4 4 (0.63%) Apache ZooKeeper
5 3 (0.48%) Apache Flume
6 1 (0.16%) VMware Infrastructure
Systems Management
1 139 (22.06%) Kubernetes
2 74 (11.75%) Terraform
3 42 (6.67%) Ansible
4 33 (5.24%) Oozie
5 30 (4.76%) Hadoop YARN
6 14 (2.22%) Puppet
7 10 (1.59%) Ambari
7 10 (1.59%) Control-M
7 10 (1.59%) Single Sign-On
8 8 (1.27%) Grafana
8 8 (1.27%) Prometheus
9 7 (1.11%) Kibana
9 7 (1.11%) Opscode Chef
10 6 (0.95%) Docker Swarm
10 6 (0.95%) logstash
11 4 (0.63%) Istio
12 2 (0.32%) Active Directory Federation Services
12 2 (0.32%) Kubeflow
12 2 (0.32%) Tivoli
12 2 (0.32%) Zipkin
Vendors
1 125 (19.84%) Databricks
2 77 (12.22%) Oracle
3 73 (11.59%) Google
4 55 (8.73%) Cloudera
5 54 (8.57%) Microsoft
6 28 (4.44%) BiZZdesign
7 25 (3.97%) Informatica
7 25 (3.97%) SAP
8 24 (3.81%) Hortonworks
9 21 (3.33%) IBM
10 16 (2.54%) Alteryx
11 13 (2.06%) Teradata
12 12 (1.90%) Ab Initio
12 12 (1.90%) MapR
13 11 (1.75%) Confluent
13 11 (1.75%) Salesforce.com
14 9 (1.43%) Adobe
14 9 (1.43%) Splunk
15 8 (1.27%) Red Hat
15 8 (1.27%) StreamSets