Period
to 17 October 2021

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 17 October 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
17 Oct 2021
Same period 2020 Same period 2019
Rank 188 201 174
Rank change year-on-year +13 -27 +29
Contract jobs citing Apache Spark 1,232 418 1,377
As % of all contract jobs advertised in the UK 1.48% 1.35% 1.62%
As % of the Application Platforms category 11.46% 9.20% 10.81%
Number of daily rates quoted 761 272 1,012
10th Percentile £425 £375 £425
25th Percentile £488 £438 £513
Median daily rate (50th Percentile) £575 £525 £550
Median % change year-on-year +9.52% -4.55% -
75th Percentile £675 £625 £638
90th Percentile £738 £700 £713
UK excluding London median daily rate £500 £475 £525
% change year-on-year +5.26% -9.52% +2.44%
Number of hourly rates quoted 11 7 2
10th Percentile £18.60 £9.75 £21.59
25th Percentile £24.50 £12.75 £22.10
Median hourly rate £27.73 £20.00 £23.33
Median % change year-on-year +38.65% -14.26% -48.17%
75th Percentile £48.13 £23.13 £24.18
90th Percentile £65.00 £24.28 £24.46
UK excluding London median hourly rate £26.12 £22.38 £23.33
% change year-on-year +16.69% -4.05% +6.02%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 10,750 4,544 12,736
As % of all contract IT jobs advertised in the UK 12.93% 14.73% 14.96%
Number of daily rates quoted 7,368 3,065 8,369
10th Percentile £300 £288 £260
25th Percentile £406 £375 £363
Median daily rate (50th Percentile) £500 £475 £450
Median % change year-on-year +5.26% +5.56% -
75th Percentile £600 £556 £550
90th Percentile £675 £650 £636
UK excluding London median daily rate £450 £446 £425
% change year-on-year +0.84% +5.00% -
Number of hourly rates quoted 166 129 296
10th Percentile £14.13 £12.63 £10.70
25th Percentile £16.19 £15.88 £14.28
Median hourly rate £24.50 £22.38 £20.92
Median % change year-on-year +9.47% +6.97% +4.61%
75th Percentile £53.75 £38.75 £40.02
90th Percentile £69.38 £58.00 £50.00
UK excluding London median hourly rate £19.75 £20.00 £20.00
% change year-on-year -1.28% - -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 17 October 2021.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 17 October 2021.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 17 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 17 October 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +14 1,042 £575 +4.55% 185
London +14 774 £600 +9.09% 136
Work from Home -19 299 £593 +8.72% 45
UK excluding London -26 269 £500 +5.26% 57
South East -30 101 £525 +10.53% 13
North of England -39 62 £565 +32.94% 9
North East - 34 £620 - 3
Scotland -52 32 £463 +12.80% 8
Wales -15 22 £550 +10.00% 3
Midlands -59 22 £463 +18.89% 10
South West -26 18 £500 -23.81% 7
Yorkshire -35 17 £450 +5.88% 4
West Midlands -55 13 £450 +12.50% 3
North West -62 12 £492 +19.33% 2
East of England -57 10 £575 -7.07% 6
East Midlands -34 8 £550 +45.50% 7
Northern Ireland -5 3 £475 +35.71% 1

For the 6 months to 17 October 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 641 (52.03%) Python
2 616 (50.00%) Hadoop
3 607 (49.27%) SQL
4 605 (49.11%) Big Data
5 505 (40.99%) Kafka
6 448 (36.36%) Scala
7 444 (36.04%) Java
8 443 (35.96%) Azure
9 423 (34.33%) AWS
10 394 (31.98%) Analytics
11 356 (28.90%) Agile Software Development
12 353 (28.65%) DevOps
13 337 (27.35%) Apache Hive
14 292 (23.70%) Data Pipeline
15 284 (23.05%) Data Modelling
16 281 (22.81%) Data Engineering
17 277 (22.48%) Finance
18 247 (20.05%) Social Skills
19 246 (19.97%) Apache
20 239 (19.40%) Kubernetes
21 235 (19.07%) Data Warehouse
22 231 (18.75%) NoSQL
23 218 (17.69%) ETL
24 216 (17.53%) Databricks
24 216 (17.53%) Data Science
25 210 (17.05%) Jenkins
26 208 (16.88%) Docker
27 200 (16.23%) Git (software)
28 190 (15.42%) Banking
29 189 (15.34%) HBase

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 246 (19.97%) Apache
2 139 (11.28%) Apache Airflow
3 82 (6.66%) Elasticsearch
4 48 (3.90%) Confluence
5 37 (3.00%) Luigi
6 24 (1.95%) Apache Pig
7 20 (1.62%) Tomcat
8 19 (1.54%) SAS
9 8 (0.65%) Jupyter
9 8 (0.65%) Mule
9 8 (0.65%) Oracle Coherence
9 8 (0.65%) SharePoint
10 6 (0.49%) GemFire
11 5 (0.41%) nginx
12 4 (0.32%) JBoss
12 4 (0.32%) Oracle Workflow
13 3 (0.24%) Apache Pulsar
13 3 (0.24%) Apache Solr
13 3 (0.24%) Camunda
14 2 (0.16%) Cloud Foundry
Applications
1 5 (0.41%) SPSS
2 4 (0.32%) Microsoft Excel
3 3 (0.24%) Microsoft PowerPoint
4 1 (0.081%) Chatbot
4 1 (0.081%) SPSS Modeler
Business Applications
1 7 (0.57%) SAP Oil and Gas
2 4 (0.32%) SAP S/4HANA
3 2 (0.16%) SAP R/3
4 1 (0.081%) Distributed Ledger
4 1 (0.081%) Oracle Cash Management
4 1 (0.081%) Oracle Procure-to-Pay
4 1 (0.081%) Towers Watson Igloo
Cloud Services
1 443 (35.96%) Azure
2 423 (34.33%) AWS
3 174 (14.12%) GCP
4 127 (10.31%) Serverless
5 122 (9.90%) Azure Data Factory
6 109 (8.85%) Amazon EMR
7 98 (7.95%) Amazon S3
8 89 (7.22%) Azure Cosmos DB
9 78 (6.33%) Azure Functions
10 74 (6.01%) GitHub
10 74 (6.01%) Power Platform
11 72 (5.84%) Amazon EC2
12 60 (4.87%) Snowflake
13 57 (4.63%) Azure Synapse Analytics
14 45 (3.65%) AWS Lambda
15 37 (3.00%) Amazon ECS
16 36 (2.92%) Azure Storage
16 36 (2.92%) Route 53
17 35 (2.84%) PaaS
18 34 (2.76%) Collibra
Communications & Networking
1 29 (2.35%) HTTP
2 24 (1.95%) DNS
3 15 (1.22%) FTP
4 14 (1.14%) Kerberos
5 12 (0.97%) Firewall
6 8 (0.65%) HTTPS
6 8 (0.65%) SSL
6 8 (0.65%) WAN
7 7 (0.57%) Network Security
8 6 (0.49%) Cisco ASA
9 5 (0.41%) SIGTRAN
9 5 (0.41%) SMPP
9 5 (0.41%) SMS
9 5 (0.41%) SS7
9 5 (0.41%) Wireshark
10 4 (0.32%) Internet
11 3 (0.24%) Intranet
11 3 (0.24%) PuTTY
12 2 (0.16%) Multicast
12 2 (0.16%) TCP/IP
Database & Business Intelligence
1 616 (50.00%) Hadoop
2 605 (49.11%) Big Data
3 337 (27.35%) Apache Hive
4 235 (19.07%) Data Warehouse
5 231 (18.75%) NoSQL
6 189 (15.34%) HBase
7 179 (14.53%) Data Lake
8 143 (11.61%) Relational Database
9 113 (9.17%) MongoDB
10 111 (9.01%) PostgreSQL
11 110 (8.93%) Apache Cassandra
12 90 (7.31%) Amazon Redshift
13 88 (7.14%) BigQuery
13 88 (7.14%) Tableau
14 86 (6.98%) Metadata
15 82 (6.66%) SQL Server
16 75 (6.09%) Impala
17 72 (5.84%) Power BI
18 66 (5.36%) Azure SQL Database
19 65 (5.28%) Flink
Development Applications
1 210 (17.05%) Jenkins
2 200 (16.23%) Git (software)
3 83 (6.74%) JIRA
4 57 (4.63%) Maven
5 54 (4.38%) GitLab
6 40 (3.25%) TeamCity
7 38 (3.08%) Hudson
8 36 (2.92%) ERwin
9 27 (2.19%) PowerDesigner
10 26 (2.11%) Subversion
11 23 (1.87%) Gradle
12 19 (1.54%) Artifactory
12 19 (1.54%) IntelliJ
13 16 (1.30%) Sonatype Nexus
14 14 (1.14%) Bitbucket
15 12 (0.97%) IBM UrbanCode
15 12 (0.97%) Visual Studio
15 12 (0.97%) Visual Studio Team System
16 11 (0.89%) Gatling
16 11 (0.89%) MLflow
General
1 277 (22.48%) Finance
2 247 (20.05%) Social Skills
3 190 (15.42%) Banking
4 147 (11.93%) Analytical Skills
5 79 (6.41%) Investment Banking
6 45 (3.65%) Presentation Skills
7 43 (3.49%) Public Sector
8 24 (1.95%) Front Office
9 19 (1.54%) Marketing
10 18 (1.46%) Financial Institution
11 17 (1.38%) Publishing
12 15 (1.22%) Retail
13 12 (0.97%) Games
13 12 (0.97%) Legal
14 11 (0.89%) Telecoms
15 9 (0.73%) Manufacturing
16 8 (0.65%) Aerospace
16 8 (0.65%) Documentation Skills
17 6 (0.49%) Retail Banking
18 4 (0.32%) Law
Job Titles
1 525 (42.61%) Data Engineer
2 238 (19.32%) Developer
3 119 (9.66%) Architect
4 91 (7.39%) Big Data Engineer
5 88 (7.14%) Azure Engineer
6 80 (6.49%) AWS Engineer
7 79 (6.41%) Java Developer
8 75 (6.09%) DevOps Engineer
9 70 (5.68%) Data Architect
10 54 (4.38%) Data Scientist
11 52 (4.22%) Senior Data Engineer
12 51 (4.14%) Scala Developer
13 43 (3.49%) Python Engineer
14 37 (3.00%) Cloud Engineer
15 34 (2.76%) Data Modeller
16 33 (2.68%) Apache Spark Developer
16 33 (2.68%) Hadoop Engineer
16 33 (2.68%) Senior Developer
17 28 (2.27%) Java Engineer
18 27 (2.19%) Big Data Developer
Libraries, Frameworks & Software Standards
1 505 (40.99%) Kafka
2 170 (13.80%) PySpark
3 133 (10.80%) Spark Streaming
4 80 (6.49%) Spring
5 55 (4.46%) JSON
6 51 (4.14%) RESTful
7 47 (3.81%) REST
8 46 (3.73%) Spring Boot
9 45 (3.65%) Web Services
10 44 (3.57%) Apache NiFi
11 34 (2.76%) ARM Templates
12 33 (2.68%) Apache Avro
12 33 (2.68%) Node.js
13 28 (2.27%) Pandas
14 23 (1.87%) .NET
14 23 (1.87%) TensorFlow
15 22 (1.79%) NumPy
16 21 (1.70%) Akka
17 20 (1.62%) AngularJS
18 19 (1.54%) XML
Miscellaneous
1 70 (5.68%) Algorithms
1 70 (5.68%) Data Structures
2 47 (3.81%) Virtual Team
3 40 (3.25%) Cloud Native
4 31 (2.52%) Data Protection Act
5 28 (2.27%) Market Risk
6 27 (2.19%) Management Information System
7 26 (2.11%) Greenfield Project
8 24 (1.95%) PKI
9 22 (1.79%) Legacy Applications
10 21 (1.70%) Public Cloud
11 19 (1.54%) iPad
12 18 (1.46%) Data Centre
13 16 (1.30%) Self-Motivation
14 15 (1.22%) Hybrid Cloud
15 13 (1.06%) Hedge funds
16 11 (0.89%) Grid Computing
16 11 (0.89%) Linux Command Line
16 11 (0.89%) Social Media
17 10 (0.81%) Legacy Systems
Operating Systems
1 123 (9.98%) Linux
2 94 (7.63%) Unix
3 24 (1.95%) Windows
4 23 (1.87%) Red Hat Enterprise Linux
5 7 (0.57%) Android
6 6 (0.49%) Debian
6 6 (0.49%) Ubuntu
7 3 (0.24%) CentOS
7 3 (0.24%) VMS
8 2 (0.16%) Apple iOS
9 1 (0.081%) DC/OS
9 1 (0.081%) Mesosphere
9 1 (0.081%) Windows Server
9 1 (0.081%) Windows Server 2012
Processes & Methodologies
1 394 (31.98%) Analytics
2 356 (28.90%) Agile Software Development
3 353 (28.65%) DevOps
4 292 (23.70%) Data Pipeline
5 284 (23.05%) Data Modelling
6 281 (22.81%) Data Engineering
7 218 (17.69%) ETL
8 216 (17.53%) Data Science
9 178 (14.45%) Machine Learning
10 168 (13.64%) Business Intelligence
11 145 (11.77%) Scrum
12 127 (10.31%) Microservices
13 116 (9.42%) Computer Science
13 116 (9.42%) Data Architecture
13 116 (9.42%) Visualisation
14 106 (8.60%) Data Management
15 103 (8.36%) Open Source
16 97 (7.87%) Data Ingestion
17 95 (7.71%) Artificial Intelligence
18 90 (7.31%) Problem-Solving
Programming Languages
1 641 (52.03%) Python
2 607 (49.27%) SQL
3 448 (36.36%) Scala
4 444 (36.04%) Java
5 89 (7.22%) R
6 56 (4.55%) JavaScript
7 42 (3.41%) C#
8 39 (3.17%) PowerShell
9 38 (3.08%) PL/SQL
10 34 (2.76%) Bash Shell
10 34 (2.76%) C++
11 32 (2.60%) T-SQL
12 31 (2.52%) Go
13 28 (2.27%) Java 8
13 28 (2.27%) Shell Script
14 25 (2.03%) Groovy
15 20 (1.62%) Perl
16 19 (1.54%) Kotlin
17 17 (1.38%) C
18 13 (1.06%) CQL
Qualifications
1 108 (8.77%) Degree
2 86 (6.98%) Security Cleared
3 65 (5.28%) SC Cleared
4 60 (4.87%) Computer Science Degree
5 22 (1.79%) BPSS Clearance
6 18 (1.46%) AWS Certification
7 13 (1.06%) Numerate Degree
8 7 (0.57%) PhD
9 5 (0.41%) DV Cleared
9 5 (0.41%) Master's Degree
10 4 (0.32%) DBS Check
10 4 (0.32%) ISEB
11 2 (0.16%) TOGAF Certification
12 1 (0.081%) CBAP
12 1 (0.081%) CCA
12 1 (0.081%) MBA
12 1 (0.081%) PMI Certification
Quality Assurance & Compliance
1 92 (7.47%) Data Quality
2 44 (3.57%) GDPR
2 44 (3.57%) QA
3 12 (0.97%) IFRS
4 7 (0.57%) Disclosure Scotland
5 5 (0.41%) MiFID
6 4 (0.32%) Sarbanes-Oxley
7 3 (0.24%) SLA
8 2 (0.16%) COBIT
8 2 (0.16%) ISO/IEC 27001
8 2 (0.16%) NIST
9 1 (0.081%) Dodd-Frank
9 1 (0.081%) EMIR
9 1 (0.081%) IFRS 9
9 1 (0.081%) ISO 20022
9 1 (0.081%) NCSC
9 1 (0.081%) PSD2
9 1 (0.081%) TARGET2
System Software
1 208 (16.88%) Docker
2 137 (11.12%) HDFS
3 29 (2.35%) Apache ZooKeeper
4 19 (1.54%) Active Directory
5 18 (1.46%) Apache Flume
6 3 (0.24%) Virtual Machines
7 2 (0.16%) VMware Infrastructure
8 1 (0.081%) Active Directory Lightweight Directory Services
8 1 (0.081%) Hyper-V
Systems Management
1 239 (19.40%) Kubernetes
2 174 (14.12%) Terraform
3 82 (6.66%) Ansible
4 57 (4.63%) Oozie
5 45 (3.65%) Prometheus
6 41 (3.33%) Hadoop YARN
7 21 (1.70%) Grafana
7 21 (1.70%) Puppet
8 13 (1.06%) Kibana
8 13 (1.06%) Opscode Chef
9 12 (0.97%) Ambari
9 12 (0.97%) Control-M
9 12 (0.97%) Single Sign-On
10 10 (0.81%) logstash
11 9 (0.73%) Kubeflow
12 8 (0.65%) Docker Swarm
13 7 (0.57%) Active Directory Federation Services
14 6 (0.49%) FortiGate
14 6 (0.49%) Rancher
15 3 (0.24%) Mesos
Vendors
1 216 (17.53%) Databricks
2 149 (12.09%) Cloudera
3 104 (8.44%) Oracle
4 101 (8.20%) Google
5 82 (6.66%) Microsoft
6 72 (5.84%) Hortonworks
7 56 (4.55%) Informatica
8 43 (3.49%) Splunk
9 33 (2.68%) IBM
9 33 (2.68%) SAP
10 32 (2.60%) BiZZdesign
11 26 (2.11%) Red Hat
12 25 (2.03%) Teradata
13 24 (1.95%) Ab Initio
14 20 (1.62%) Confluent
14 20 (1.62%) Salesforce.com
15 18 (1.46%) StreamSets
16 17 (1.38%) Alteryx
16 17 (1.38%) MapR
17 15 (1.22%) Talend