Period
to 21 October 2021

The following table provides summary statistics for contract job vacancies advertised in London with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 21 October 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
England > London
6 months to
21 Oct 2021
Same period 2020 Same period 2019
Rank 128 135 103
Rank change year-on-year +7 -32 +40
Contract jobs citing Apache Spark 787 286 929
As % of all contract jobs advertised in London 2.19% 2.03% 2.52%
As % of the Application Platforms category 16.50% 13.81% 16.31%
Number of daily rates quoted 515 190 742
10th Percentile £450 £413 £457
25th Percentile £525 £463 £513
Median daily rate (50th Percentile) £600 £550 £575
Median % change year-on-year +9.09% -4.35% -
75th Percentile £688 £629 £675
90th Percentile £750 £713 £725
England median daily rate £575 £550 £563
% change year-on-year +4.55% -2.22% +2.27%
Number of hourly rates quoted 5 0 0
Median hourly rate £55.00 - -
England median hourly rate £37.50 £19.44 £23.33
% change year-on-year +92.90% -16.66% -30.89%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies advertised in London with a requirement for application platform skills.

All Application Platform Skills
London
Contract vacancies with a requirement for application platform skills 4,769 2,071 5,695
As % of all contract IT jobs advertised in London 13.29% 14.71% 15.46%
Number of daily rates quoted 3,378 1,455 4,091
10th Percentile £350 £313 £294
25th Percentile £450 £413 £406
Median daily rate (50th Percentile) £550 £500 £500
Median % change year-on-year +10.00% - -
75th Percentile £650 £595 £588
90th Percentile £720 £700 £675
England median daily rate £500 £475 £463
% change year-on-year +5.26% +2.70% -
Number of hourly rates quoted 31 16 50
10th Percentile £17.62 £15.00 £15.23
25th Percentile £22.75 £22.56 £17.00
Median hourly rate £32.50 £36.25 £21.00
Median % change year-on-year -10.34% +72.62% +5.00%
75th Percentile £53.75 £41.63 £36.70
90th Percentile £70.00 £48.38 £50.00
England median hourly rate £22.50 £22.50 £19.50
% change year-on-year - +15.38% -2.50%

Apache Spark
Job Vacancy Trend in London

Job postings citing Apache Spark as a proportion of all IT jobs advertised in London.

Job vacancy trend for Apache Spark in London

Apache Spark
Contractor Daily Rate Trend in London

3-month moving average daily rate quoted in jobs citing Apache Spark in London.

Daily rate trend for Apache Spark in London

Apache Spark
Daily Rate Histogram in London

Daily rate distribution for jobs citing Apache Spark in London over the 6 months to 21 October 2021.

Daily rate histogram for Apache Spark in London

Apache Spark
Contractor Hourly Rate Trend in London

3-month moving average hourly rates quoted in jobs citing Apache Spark in London.

Hourly rate trend for Apache Spark in London

Apache Spark
Hourly Rate Histogram in London

Hourly rate distribution of jobs citing Apache Spark in London over the 6 months to 21 October 2021.

Hourly rate histogram for Apache Spark in London

Apache Spark
Contract Job Locations in London

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the London region over the 6 months to 21 October 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
Central London -30 84 £615 +17.10% 42
South London +8 11 £625 +16.82% 1
West London -32 4 £603 +4.78% 8
East London - 1 - - 6
North London - 1 - - 5
Apache Spark
England

For the 6 months to 21 October 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads across the London region with a requirement for Apache Spark.

1 421 (53.49%) Big Data
2 410 (52.10%) Python
3 393 (49.94%) SQL
4 354 (44.98%) Hadoop
5 317 (40.28%) Kafka
6 304 (38.63%) Azure
7 298 (37.87%) Scala
8 290 (36.85%) Java
9 279 (35.45%) Analytics
10 245 (31.13%) Agile Software Development
11 243 (30.88%) AWS
12 207 (26.30%) Finance
13 205 (26.05%) DevOps
14 199 (25.29%) Apache Hive
15 196 (24.90%) Data Pipeline
16 177 (22.49%) Data Modelling
17 174 (22.11%) Banking
18 165 (20.97%) Kubernetes
19 163 (20.71%) Social Skills
20 158 (20.08%) NoSQL
21 156 (19.82%) Data Engineering
22 155 (19.70%) Data Science
23 153 (19.44%) Apache
24 143 (18.17%) Jenkins
25 140 (17.79%) Data Warehouse
26 136 (17.28%) Docker
26 136 (17.28%) Databricks
27 129 (16.39%) ETL
27 129 (16.39%) Machine Learning
28 126 (16.01%) GCP

Apache Spark
Co-occurring IT Skills in London by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 153 (19.44%) Apache
2 99 (12.58%) Apache Airflow
3 46 (5.84%) Elasticsearch
4 39 (4.96%) Confluence
5 31 (3.94%) Luigi
6 19 (2.41%) Tomcat
7 17 (2.16%) Apache Pig
8 10 (1.27%) SAS
9 7 (0.89%) Jupyter
9 7 (0.89%) Oracle Coherence
10 5 (0.64%) Mule
10 5 (0.64%) SharePoint
11 4 (0.51%) Camunda
11 4 (0.51%) GemFire
12 3 (0.38%) JBoss
13 2 (0.25%) Cloud Foundry
13 2 (0.25%) Oracle Workflow
14 1 (0.13%) ArcGIS
14 1 (0.13%) OpenStack
Applications
1 5 (0.64%) SPSS
2 3 (0.38%) Microsoft Excel
3 2 (0.25%) Microsoft PowerPoint
4 1 (0.13%) SPSS Modeler
Business Applications
1 6 (0.76%) SAP Oil and Gas
2 4 (0.51%) SAP S/4HANA
3 1 (0.13%) Distributed Ledger
3 1 (0.13%) Oracle Cash Management
3 1 (0.13%) SAP R/3
3 1 (0.13%) Towers Watson Igloo
Cloud Services
1 304 (38.63%) Azure
2 243 (30.88%) AWS
3 126 (16.01%) GCP
4 83 (10.55%) Serverless
5 82 (10.42%) Azure Data Factory
6 65 (8.26%) Azure Cosmos DB
7 61 (7.75%) Amazon S3
8 60 (7.62%) Azure Functions
9 56 (7.12%) Amazon EMR
10 46 (5.84%) Power Platform
11 42 (5.34%) GitHub
12 36 (4.57%) Amazon EC2
13 32 (4.07%) Azure Synapse Analytics
14 29 (3.68%) Collibra
15 24 (3.05%) Snowflake
16 23 (2.92%) AWS Lambda
17 18 (2.29%) PaaS
18 17 (2.16%) Azure Storage
19 16 (2.03%) Route 53
20 15 (1.91%) IaaS
Communications & Networking
1 9 (1.14%) HTTP
2 8 (1.02%) FTP
3 7 (0.89%) Firewall
3 7 (0.89%) Kerberos
4 6 (0.76%) DNS
5 3 (0.38%) HTTPS
5 3 (0.38%) Intranet
5 3 (0.38%) VPN
5 3 (0.38%) WAN
6 2 (0.25%) LAN
6 2 (0.25%) Multicast
6 2 (0.25%) SSL
6 2 (0.25%) TCP/IP
7 1 (0.13%) Internet
7 1 (0.13%) IPv4
7 1 (0.13%) IPv6
7 1 (0.13%) SMPP
7 1 (0.13%) SMTP
7 1 (0.13%) SNMP
7 1 (0.13%) SS7
Database & Business Intelligence
1 421 (53.49%) Big Data
2 354 (44.98%) Hadoop
3 199 (25.29%) Apache Hive
4 158 (20.08%) NoSQL
5 140 (17.79%) Data Warehouse
6 111 (14.10%) Data Lake
7 105 (13.34%) HBase
8 85 (10.80%) Relational Database
9 81 (10.29%) Apache Cassandra
10 79 (10.04%) PostgreSQL
11 72 (9.15%) MongoDB
12 70 (8.89%) BigQuery
13 66 (8.39%) Tableau
14 61 (7.75%) SQL Server
15 50 (6.35%) Amazon Redshift
16 46 (5.84%) Power BI
17 45 (5.72%) Metadata
18 41 (5.21%) Spark SQL
19 37 (4.70%) Flink
19 37 (4.70%) Impala
Development Applications
1 143 (18.17%) Jenkins
2 122 (15.50%) Git (software)
3 66 (8.39%) JIRA
4 47 (5.97%) GitLab
5 43 (5.46%) Maven
6 38 (4.83%) Hudson
7 36 (4.57%) TeamCity
8 27 (3.43%) ERwin
9 26 (3.30%) PowerDesigner
10 25 (3.18%) Subversion
11 17 (2.16%) Gradle
11 17 (2.16%) IntelliJ
12 11 (1.40%) IBM UrbanCode
13 10 (1.27%) MLflow
13 10 (1.27%) Visual Studio Team System
14 8 (1.02%) Bitbucket
15 7 (0.89%) Artifactory
15 7 (0.89%) Cucumber
15 7 (0.89%) Mockito
15 7 (0.89%) Sonatype Nexus
General
1 207 (26.30%) Finance
2 174 (22.11%) Banking
3 163 (20.71%) Social Skills
4 101 (12.83%) Analytical Skills
5 82 (10.42%) Investment Banking
6 36 (4.57%) Presentation Skills
7 23 (2.92%) Public Sector
8 22 (2.80%) Front Office
9 16 (2.03%) Marketing
10 15 (1.91%) Financial Institution
11 12 (1.52%) Legal
12 9 (1.14%) Games
12 9 (1.14%) Publishing
12 9 (1.14%) Retail
13 6 (0.76%) Aerospace
14 4 (0.51%) Back Office
14 4 (0.51%) Law
15 3 (0.38%) Retail Banking
15 3 (0.38%) Telecoms
16 2 (0.25%) Electronics
Job Titles
1 319 (40.53%) Data Engineer
2 180 (22.87%) Developer
3 82 (10.42%) Architect
4 70 (8.89%) Java Developer
5 68 (8.64%) Big Data Engineer
6 49 (6.23%) Azure Engineer
7 48 (6.10%) Data Architect
8 46 (5.84%) Scala Developer
9 37 (4.70%) DevOps Engineer
10 32 (4.07%) Senior Data Engineer
11 31 (3.94%) AWS Engineer
11 31 (3.94%) Data Scientist
12 30 (3.81%) Apache Spark Developer
13 28 (3.56%) Senior Developer
14 25 (3.18%) Data Modeller
15 23 (2.92%) Data Manager
16 22 (2.80%) Cloud Engineer
17 21 (2.67%) Azure Architect
17 21 (2.67%) Hadoop Engineer
18 19 (2.41%) Big Data Developer
Libraries, Frameworks & Software Standards
1 317 (40.28%) Kafka
2 118 (14.99%) PySpark
3 67 (8.51%) Spring
4 64 (8.13%) Spark Streaming
5 39 (4.96%) Spring Boot
6 36 (4.57%) RESTful
7 26 (3.30%) ARM Templates
8 25 (3.18%) Apache Avro
9 23 (2.92%) JSON
10 21 (2.67%) REST
11 19 (2.41%) Akka
11 19 (2.41%) Web Services
12 18 (2.29%) Pandas
13 17 (2.16%) Node.js
14 16 (2.03%) Velocity
15 15 (1.91%) Apache NiFi
15 15 (1.91%) Elastic Stack
15 15 (1.91%) NumPy
16 14 (1.78%) OAuth
17 13 (1.65%) XML
Miscellaneous
1 64 (8.13%) Algorithms
2 43 (5.46%) Data Structures
2 43 (5.46%) Virtual Team
3 25 (3.18%) Market Risk
4 22 (2.80%) Cloud Native
5 20 (2.54%) Management Information System
6 19 (2.41%) Data Protection Act
6 19 (2.41%) Legacy Applications
7 16 (2.03%) Greenfield Project
8 15 (1.91%) Data Centre
8 15 (1.91%) Self-Motivation
9 10 (1.27%) Foreign Exchange (FX)
9 10 (1.27%) Hedge funds
9 10 (1.27%) Public Cloud
10 7 (0.89%) Equities
10 7 (0.89%) Fixed Income
10 7 (0.89%) Grid Computing
10 7 (0.89%) Legacy Systems
10 7 (0.89%) Linux Command Line
11 6 (0.76%) Replication
Operating Systems
1 65 (8.26%) Linux
2 54 (6.86%) Unix
3 10 (1.27%) Windows
4 8 (1.02%) Red Hat Enterprise Linux
5 3 (0.38%) Android
5 3 (0.38%) Debian
5 3 (0.38%) Ubuntu
5 3 (0.38%) VMS
6 2 (0.25%) Apple iOS
7 1 (0.13%) Windows Server
7 1 (0.13%) Windows Server 2012
Processes & Methodologies
1 279 (35.45%) Analytics
2 245 (31.13%) Agile Software Development
3 205 (26.05%) DevOps
4 196 (24.90%) Data Pipeline
5 177 (22.49%) Data Modelling
6 156 (19.82%) Data Engineering
7 155 (19.70%) Data Science
8 129 (16.39%) ETL
8 129 (16.39%) Machine Learning
9 120 (15.25%) Business Intelligence
10 105 (13.34%) Scrum
11 91 (11.56%) Microservices
12 87 (11.05%) Visualisation
13 85 (10.80%) Data Architecture
14 84 (10.67%) Data Management
15 83 (10.55%) Computer Science
16 78 (9.91%) Artificial Intelligence
17 71 (9.02%) Open Source
18 62 (7.88%) Data Integration
19 60 (7.62%) Use Case
Programming Languages
1 410 (52.10%) Python
2 393 (49.94%) SQL
3 298 (37.87%) Scala
4 290 (36.85%) Java
5 57 (7.24%) R
6 34 (4.32%) JavaScript
7 29 (3.68%) C#
8 27 (3.43%) PL/SQL
9 24 (3.05%) C++
10 21 (2.67%) Go
10 21 (2.67%) Java 8
11 19 (2.41%) PowerShell
11 19 (2.41%) Shell Script
12 18 (2.29%) Bash Shell
12 18 (2.29%) T-SQL
13 17 (2.16%) Kotlin
14 16 (2.03%) C
15 15 (1.91%) Perl
16 10 (1.27%) CQL
17 9 (1.14%) Groovy
Qualifications
1 76 (9.66%) Degree
2 45 (5.72%) Computer Science Degree
3 25 (3.18%) Security Cleared
4 18 (2.29%) SC Cleared
5 13 (1.65%) AWS Certification
6 11 (1.40%) Numerate Degree
7 5 (0.64%) BPSS Clearance
8 3 (0.38%) DBS Check
8 3 (0.38%) ISEB
8 3 (0.38%) Master's Degree
8 3 (0.38%) PhD
9 2 (0.25%) DV Cleared
10 1 (0.13%) CBAP
10 1 (0.13%) CCA
10 1 (0.13%) MBA
10 1 (0.13%) PMI Certification
Quality Assurance & Compliance
1 60 (7.62%) Data Quality
2 39 (4.96%) GDPR
3 22 (2.80%) QA
4 9 (1.14%) IFRS
5 5 (0.64%) MiFID
6 2 (0.25%) COBIT
6 2 (0.25%) ISO/IEC 27001
6 2 (0.25%) NIST
6 2 (0.25%) Sarbanes-Oxley
7 1 (0.13%) Dodd-Frank
7 1 (0.13%) EMIR
7 1 (0.13%) IFRS 9
7 1 (0.13%) ISO 20022
7 1 (0.13%) NCSC
7 1 (0.13%) PSD2
7 1 (0.13%) TARGET2
System Software
1 136 (17.28%) Docker
2 64 (8.13%) HDFS
3 10 (1.27%) Apache ZooKeeper
4 7 (0.89%) Active Directory
5 6 (0.76%) Apache Flume
6 2 (0.25%) Active Directory Lightweight Directory Services
7 1 (0.13%) VMware Infrastructure
Systems Management
1 165 (20.97%) Kubernetes
2 86 (10.93%) Terraform
3 59 (7.50%) Ansible
4 36 (4.57%) Oozie
5 25 (3.18%) Hadoop YARN
6 18 (2.29%) Prometheus
7 17 (2.16%) Puppet
8 12 (1.52%) Single Sign-On
9 11 (1.40%) Control-M
10 10 (1.27%) Kibana
11 8 (1.02%) Docker Swarm
11 8 (1.02%) Kubeflow
11 8 (1.02%) logstash
11 8 (1.02%) Opscode Chef
12 7 (0.89%) Grafana
13 5 (0.64%) Ambari
14 3 (0.38%) Istio
15 2 (0.25%) Active Directory Federation Services
15 2 (0.25%) Consul
15 2 (0.25%) Zipkin
Vendors
1 136 (17.28%) Databricks
2 86 (10.93%) Cloudera
3 73 (9.28%) Google
4 72 (9.15%) Oracle
5 59 (7.50%) Hortonworks
6 53 (6.73%) Microsoft
7 34 (4.32%) IBM
8 30 (3.81%) BiZZdesign
9 27 (3.43%) Informatica
10 26 (3.30%) SAP
11 24 (3.05%) Splunk
12 19 (2.41%) Ab Initio
13 17 (2.16%) MapR
13 17 (2.16%) Teradata
14 16 (2.03%) Alteryx
15 14 (1.78%) Red Hat
15 14 (1.78%) Salesforce.com
16 10 (1.27%) AppDynamics
17 9 (1.14%) Netezza
18 8 (1.02%) Confluent