Period
to 10 May 2021

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 10 May 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
10 May 2021
Same period 2020 Same period 2019
Rank 172 201 159
Rank change year-on-year +29 -42 +21
Contract jobs citing Apache Spark 843 689 1,668
As % of all contract jobs advertised in the UK 1.51% 1.33% 1.80%
As % of the Application Platforms category 11.20% 9.18% 11.39%
Number of daily rates quoted 526 508 1,194
10th Percentile £425 £413 £443
25th Percentile £490 £463 £500
Median daily rate (50th Percentile) £563 £550 £550
Median % change year-on-year +2.27% - -
75th Percentile £638 £625 £625
90th Percentile £700 £683 £713
UK excluding London median daily rate £508 £500 £515
% change year-on-year +1.50% -2.91% +3.00%
Number of hourly rates quoted 4 0 1
10th Percentile £17.10 " "
25th Percentile £21.38 " £52.50
Median hourly rate £25.75 - £55.00
75th Percentile £39.63 " £57.50
90th Percentile £62.35 " "
UK excluding London median hourly rate £24.50 - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 7,530 7,508 14,640
As % of all contract IT jobs advertised in the UK 13.52% 14.46% 15.76%
Number of daily rates quoted 5,136 4,960 9,477
10th Percentile £300 £263 £281
25th Percentile £400 £363 £375
Median daily rate (50th Percentile) £475 £473 £475
Median % change year-on-year +0.53% -0.53% +5.56%
75th Percentile £575 £563 £575
90th Percentile £650 £650 £638
UK excluding London median daily rate £445 £438 £438
% change year-on-year +1.71% - +6.06%
Number of hourly rates quoted 160 171 279
10th Percentile £12.25 £12.63 £11.95
25th Percentile £17.13 £15.00 £12.50
Median hourly rate £25.25 £20.00 £18.00
Median % change year-on-year +26.25% +11.11% +2.86%
75th Percentile £52.63 £35.50 £37.50
90th Percentile £65.03 £53.75 £48.90
UK excluding London median hourly rate £24.00 £19.00 £16.36
% change year-on-year +26.32% +16.17% -3.79%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 10 May 2021.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 10 May 2021.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 10 May 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +37 696 £575 +4.55% 173
London +19 494 £580 +5.45% 132
Work from Home -71 241 £553 -7.92% 26
UK excluding London +8 194 £508 +1.50% 48
South East +18 56 £506 +4.11% 13
North of England +42 45 £555 +0.91% 11
South West +46 33 £500 -8.05% 2
North West +22 24 £594 +7.95% 9
Yorkshire +18 19 £553 +16.32% 2
Midlands +24 17 £425 -19.05% 4
Scotland -17 15 £525 -8.70% 6
East of England +19 14 £350 -29.11% 9
Wales -6 14 £550 +10.00%
West Midlands +4 12 £425 -19.05% 4
North East +26 2 £605 -6.92%
East Midlands +17 2 £538 -

For the 6 months to 10 May 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 439 (52.08%) Python
2 435 (51.60%) Hadoop
3 428 (50.77%) Big Data
4 362 (42.94%) Scala
5 355 (42.11%) SQL
6 323 (38.32%) Kafka
7 320 (37.96%) Java
8 274 (32.50%) Apache Hive
9 271 (32.15%) Agile Software Development
10 264 (31.32%) Azure
11 253 (30.01%) Apache
12 236 (28.00%) Finance
13 235 (27.88%) Analytics
14 229 (27.16%) AWS
15 203 (24.08%) DevOps
16 180 (21.35%) Data Pipeline
17 168 (19.93%) Data Engineering
18 156 (18.51%) Kubernetes
19 153 (18.15%) Data Science
20 151 (17.91%) Social Skills
21 148 (17.56%) NoSQL
22 145 (17.20%) ETL
23 142 (16.84%) Machine Learning
24 138 (16.37%) Banking
25 134 (15.90%) Jenkins
26 132 (15.66%) Data Warehouse
27 131 (15.54%) Data Lake
28 130 (15.42%) Data Modelling
29 129 (15.30%) Databricks
30 126 (14.95%) Analytical Skills

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 253 (30.01%) Apache
2 121 (14.35%) Apache Airflow
3 58 (6.88%) Confluence
4 42 (4.98%) Elasticsearch
5 27 (3.20%) Oracle Coherence
6 12 (1.42%) SAS
6 12 (1.42%) SharePoint
7 11 (1.30%) Apache Pig
7 11 (1.30%) Luigi
8 9 (1.07%) Apache Solr
9 8 (0.95%) Apache Pulsar
10 7 (0.83%) GemFire
11 5 (0.59%) Jupyter
12 4 (0.47%) JBoss
12 4 (0.47%) Oracle Workflow
13 3 (0.36%) ArcGIS
14 1 (0.12%) DataSynapse
Applications
1 5 (0.59%) Stata
2 3 (0.36%) Microsoft PowerPoint
2 3 (0.36%) SPSS
3 1 (0.12%) Microsoft Excel
3 1 (0.12%) Microsoft Office
3 1 (0.12%) MS Visio
Business Applications
1 5 (0.59%) SAP Oil and Gas
2 2 (0.24%) Temenos T24
3 1 (0.12%) SAP CRM
3 1 (0.12%) SAP ERP
3 1 (0.12%) SAP IS-U
Cloud Services
1 264 (31.32%) Azure
2 229 (27.16%) AWS
3 122 (14.47%) GCP
4 80 (9.49%) Amazon S3
5 69 (8.19%) Serverless
6 51 (6.05%) Azure Data Factory
7 49 (5.81%) Amazon EMR
8 48 (5.69%) GitHub
9 47 (5.58%) Azure Cosmos DB
10 43 (5.10%) Power Platform
11 39 (4.63%) Azure Functions
12 35 (4.15%) Azure Synapse Analytics
13 31 (3.68%) Amazon EC2
14 30 (3.56%) AWS Lambda
15 25 (2.97%) Amazon Kinesis
16 21 (2.49%) AWS CloudFormation
17 18 (2.14%) Google Analytics
18 17 (2.02%) Cloud Computing
19 15 (1.78%) Azure Storage
20 14 (1.66%) Amazon SageMaker
Communications & Networking
1 24 (2.85%) HTTP
2 19 (2.25%) Kerberos
3 9 (1.07%) Firewall
3 9 (1.07%) SSL
4 8 (0.95%) Network Security
5 6 (0.71%) DNS
6 3 (0.36%) SIGTRAN
6 3 (0.36%) SMPP
6 3 (0.36%) SMS
6 3 (0.36%) SS7
6 3 (0.36%) Wireshark
7 2 (0.24%) Internet
8 1 (0.12%) ADSL
8 1 (0.12%) FTP
8 1 (0.12%) HTTPS
8 1 (0.12%) Software-Defined Networking
Database & Business Intelligence
1 435 (51.60%) Hadoop
2 428 (50.77%) Big Data
3 274 (32.50%) Apache Hive
4 148 (17.56%) NoSQL
5 132 (15.66%) Data Warehouse
6 131 (15.54%) Data Lake
7 117 (13.88%) HBase
8 86 (10.20%) MongoDB
9 84 (9.96%) Relational Database
10 81 (9.61%) Impala
11 60 (7.12%) PostgreSQL
12 56 (6.64%) Amazon Redshift
13 54 (6.41%) Spark SQL
14 51 (6.05%) BigQuery
15 50 (5.93%) SQL Server
16 48 (5.69%) Snowflake Schema
17 47 (5.58%) Apache Cassandra
18 45 (5.34%) Tableau
19 44 (5.22%) Amazon Athena
20 43 (5.10%) Parquet
Development Applications
1 134 (15.90%) Jenkins
2 114 (13.52%) Git (software)
3 67 (7.95%) JIRA
4 43 (5.10%) Sonatype Nexus
5 38 (4.51%) Maven
6 26 (3.08%) IBM UrbanCode
7 24 (2.85%) GitLab
8 22 (2.61%) TeamCity
9 16 (1.90%) Gradle
10 14 (1.66%) Artifactory
10 14 (1.66%) RStudio
11 11 (1.30%) Oracle APEX
12 10 (1.19%) JUnit
12 10 (1.19%) Visual Studio Team System
13 9 (1.07%) Cucumber
14 8 (0.95%) Bitbucket
14 8 (0.95%) MLflow
14 8 (0.95%) PowerDesigner
15 7 (0.83%) Hudson
15 7 (0.83%) Selenium
General
1 236 (28.00%) Finance
2 151 (17.91%) Social Skills
3 138 (16.37%) Banking
4 126 (14.95%) Analytical Skills
5 65 (7.71%) Front Office
6 54 (6.41%) Investment Banking
7 25 (2.97%) Public Sector
8 24 (2.85%) Marketing
9 21 (2.49%) Presentation Skills
10 16 (1.90%) Retail Banking
11 14 (1.66%) Retail
12 10 (1.19%) Financial Institution
13 8 (0.95%) Games
14 7 (0.83%) Publishing
15 6 (0.71%) Back Office
16 5 (0.59%) French Language
16 5 (0.59%) Influencing Skills
16 5 (0.59%) Organisational Skills
17 4 (0.47%) Telecoms
18 3 (0.36%) Legal
Job Titles
1 285 (33.81%) Data Engineer
2 267 (31.67%) Developer
3 98 (11.63%) Scala Developer
4 81 (9.61%) Architect
5 76 (9.02%) Java Developer
6 68 (8.07%) Big Data Engineer
7 58 (6.88%) Apache Spark Developer
8 56 (6.64%) Data Scientist
9 51 (6.05%) Data Architect
10 50 (5.93%) AWS Engineer
10 50 (5.93%) Hadoop Developer
11 49 (5.81%) Senior Developer
12 46 (5.46%) Big Data Developer
13 45 (5.34%) Scala Engineer
14 41 (4.86%) Python Engineer
15 39 (4.63%) Azure Engineer
16 34 (4.03%) Software Engineer
17 25 (2.97%) Analyst
17 25 (2.97%) Python Developer
18 24 (2.85%) Senior Java Developer
Libraries, Frameworks & Software Standards
1 323 (38.32%) Kafka
2 93 (11.03%) PySpark
3 64 (7.59%) Spark Streaming
4 57 (6.76%) Web Services
5 42 (4.98%) JSON
6 40 (4.74%) Spring
7 33 (3.91%) RESTful
8 31 (3.68%) Node.js
9 25 (2.97%) NumPy
10 21 (2.49%) Apache NiFi
10 21 (2.49%) Pandas
10 21 (2.49%) TensorFlow
10 21 (2.49%) XML
11 20 (2.37%) Apache Avro
12 19 (2.25%) PyTorch
13 18 (2.14%) React
13 18 (2.14%) Velocity
14 17 (2.02%) HTML
14 17 (2.02%) Protocol Buffers
14 17 (2.02%) Samza
Miscellaneous
1 63 (7.47%) Algorithms
2 49 (5.81%) Data Structures
3 34 (4.03%) Linux Command Line
4 32 (3.80%) Market Risk
5 31 (3.68%) Grid Computing
6 23 (2.73%) Greenfield Project
6 23 (2.73%) Virtual Team
7 22 (2.61%) Equities
7 22 (2.61%) Self-Motivation
8 21 (2.49%) iPad
9 17 (2.02%) Fixed Income
10 15 (1.78%) Derivative
11 12 (1.42%) Foreign Exchange (FX)
12 10 (1.19%) Management Information System
13 8 (0.95%) Enterprise Software
13 8 (0.95%) Hedge funds
13 8 (0.95%) Replication
14 7 (0.83%) Analytical Mindset
14 7 (0.83%) Cloud Native
14 7 (0.83%) User Experience
Operating Systems
1 103 (12.22%) Linux
2 34 (4.03%) Unix
3 14 (1.66%) Windows
4 8 (0.95%) Red Hat Enterprise Linux
5 5 (0.59%) Android
6 1 (0.12%) Apple iOS
6 1 (0.12%) CentOS
6 1 (0.12%) Mac OS X
6 1 (0.12%) Windows Server
Processes & Methodologies
1 271 (32.15%) Agile Software Development
2 235 (27.88%) Analytics
3 203 (24.08%) DevOps
4 180 (21.35%) Data Pipeline
5 168 (19.93%) Data Engineering
6 153 (18.15%) Data Science
7 145 (17.20%) ETL
8 142 (16.84%) Machine Learning
9 130 (15.42%) Data Modelling
10 100 (11.86%) Scrum
10 100 (11.86%) TDD
11 98 (11.63%) Business Intelligence
12 95 (11.27%) Software Engineering
13 85 (10.08%) Computer Science
14 79 (9.37%) BDD
15 77 (9.13%) Data Architecture
15 77 (9.13%) Mathematics
16 74 (8.78%) Test Automation
17 73 (8.66%) Data Ingestion
18 67 (7.95%) Migration
Programming Languages
1 439 (52.08%) Python
2 362 (42.94%) Scala
3 355 (42.11%) SQL
4 320 (37.96%) Java
5 74 (8.78%) R
6 46 (5.46%) JavaScript
7 36 (4.27%) C#
7 36 (4.27%) Shell Script
8 28 (3.32%) C++
9 22 (2.61%) Haskell
10 21 (2.49%) PL/SQL
11 20 (2.37%) T-SQL
12 16 (1.90%) Bash Shell
12 16 (1.90%) Go
13 14 (1.66%) Kotlin
14 13 (1.54%) Groovy
14 13 (1.54%) Java 8
15 11 (1.30%) Elm
16 7 (0.83%) C
17 4 (0.47%) Ruby
Qualifications
1 79 (9.37%) Degree
2 66 (7.83%) Security Cleared
3 60 (7.12%) SC Cleared
4 27 (3.20%) Numerate Degree
5 19 (2.25%) Computer Science Degree
6 10 (1.19%) AWS Certification
6 10 (1.19%) ISEB
6 10 (1.19%) PhD
7 8 (0.95%) Master's Degree
8 7 (0.83%) BPSS Clearance
9 3 (0.36%) DBS Check
10 1 (0.12%) TOGAF Certification
Quality Assurance & Compliance
1 45 (5.34%) Data Quality
2 40 (4.74%) QA
3 17 (2.02%) GDPR
4 3 (0.36%) SLA
5 2 (0.24%) Disclosure Scotland
5 2 (0.24%) MiFID
6 1 (0.12%) Basel II
6 1 (0.12%) ISO 20022
6 1 (0.12%) PCI DSS
6 1 (0.12%) SEPA
6 1 (0.12%) TARGET2
System Software
1 121 (14.35%) Docker
2 118 (14.00%) HDFS
3 18 (2.14%) Apache ZooKeeper
4 17 (2.02%) Apache Flume
5 7 (0.83%) Virtual Machines
6 4 (0.47%) Active Directory
Systems Management
1 156 (18.51%) Kubernetes
2 79 (9.37%) Terraform
3 69 (8.19%) Oozie
4 61 (7.24%) Ansible
5 44 (5.22%) Hadoop YARN
6 18 (2.14%) Puppet
7 10 (1.19%) Control-M
7 10 (1.19%) Kibana
8 8 (0.95%) Ambari
9 7 (0.83%) Grafana
9 7 (0.83%) Opscode Chef
9 7 (0.83%) Prometheus
10 3 (0.36%) Apache Slider
11 2 (0.24%) Istio
11 2 (0.24%) Kubeflow
11 2 (0.24%) Mesos
12 1 (0.12%) Chronos
12 1 (0.12%) Marathon
12 1 (0.12%) Rancher
12 1 (0.12%) Zipkin
Vendors
1 129 (15.30%) Databricks
2 98 (11.63%) Oracle
3 82 (9.73%) Cloudera
4 70 (8.30%) Google
5 60 (7.12%) Microsoft
6 26 (3.08%) Confluent
6 26 (3.08%) Informatica
7 23 (2.73%) Hortonworks
8 19 (2.25%) Teradata
9 17 (2.02%) SAP
10 16 (1.90%) Talend
11 15 (1.78%) Alteryx
12 13 (1.54%) BiZZdesign
12 13 (1.54%) MapR
13 12 (1.42%) Bloomberg
13 12 (1.42%) StreamSets
14 9 (1.07%) Adobe
15 7 (0.83%) Ab Initio
15 7 (0.83%) Red Hat
15 7 (0.83%) Splunk