Period
to 2 March 2021

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 2 March 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
2 Mar 2021
Same period 2020 Same period 2019
Rank 184 204 189
Rank change year-on-year +20 -15 -6
Contract jobs citing Apache Spark 661 860 1,464
As % of all contract jobs advertised in the UK 1.38% 1.29% 1.49%
As % of the Application Platforms category 9.79% 9.02% 9.24%
Number of daily rates quoted 581 656 1,068
Median daily rate £555 £550 £550
Median daily rate % change year-on-year +0.91% - -
10th Percentile £413 £413 £425
90th Percentile £688 £725 £713
UK excluding London median daily rate £500 £525 £525
% change year-on-year -4.76% - +5.00%
Number of hourly rates quoted 4 0 3
Median hourly rate £23.77 - £45.00
10th Percentile £21.59 - £42.50
90th Percentile £61.77 - £55.50
UK excluding London median hourly rate £25.05 - -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 6,755 9,532 15,843
As % of all contract IT jobs advertised in the UK 14.05% 14.33% 16.16%
Number of daily rates quoted 4,591 6,289 10,417
Median daily rate £475 £450 £475
Median daily rate % change year-on-year +5.56% -5.26% +5.56%
10th Percentile £280 £255 £280
90th Percentile £650 £638 £638
UK excluding London median daily rate £429 £425 £425
% change year-on-year +0.94% - +6.25%
Number of hourly rates quoted 186 193 305
Median hourly rate £32.00 £19.44 £18.50
Median hourly rate % change year-on-year +64.61% +5.08% -
10th Percentile £13.77 £11.52 £12.00
90th Percentile £64.74 £53.00 £47.70
Median hourly rate £24.00 £18.50 £17.13
% change year-on-year +29.73% +8.03% -2.14%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 2 March 2021.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 2 March 2021.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 15 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 2 March 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +18 569 £570 +2.24% 155
London +16 424 £575 -2.75% 113
Work from Home -92 171 £553 -19.93% 54
UK excluding London -3 134 £500 -4.76% 49
South East +28 41 £500 - 13
North of England +58 33 £550 - 20
South West +55 20 £500 -9.09% 3
North West +42 18 £485 -11.82% 8
East of England +42 17 £350 -30.00% 2
Yorkshire +31 15 £550 +3.77% 11
Wales -4 8 £538 +8.59% 2
Midlands +22 7 £375 -28.57% 4
West Midlands +17 7 £375 -28.57% 4
Scotland -10 6 £379 -34.09% 5
Northern Ireland - 2 £450 -

For the 6 months to 2 March 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 384 (58.09%) Python
2 372 (56.28%) Hadoop
3 335 (50.68%) Big Data
4 268 (40.54%) Scala
5 261 (39.49%) Java
6 256 (38.73%) SQL
7 209 (31.62%) AWS
8 205 (31.01%) Apache Hive
9 204 (30.86%) Agile Software Development
10 200 (30.26%) Kafka
11 196 (29.65%) Apache
12 194 (29.35%) Finance
13 139 (21.03%) Analytics
13 139 (21.03%) Azure
14 135 (20.42%) Machine Learning
15 128 (19.36%) Data Science
16 127 (19.21%) DevOps
17 124 (18.76%) ETL
18 117 (17.70%) HDFS
19 106 (16.04%) Banking
20 98 (14.83%) Kubernetes
21 96 (14.52%) Data Engineering
21 96 (14.52%) Data Lake
22 95 (14.37%) NoSQL
23 94 (14.22%) Data Warehouse
24 92 (13.92%) Docker
25 91 (13.77%) Jenkins
26 89 (13.46%) Analytical Skills
26 89 (13.46%) Data Modelling
27 88 (13.31%) Software Engineering

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 196 (29.65%) Apache
2 63 (9.53%) Apache Airflow
3 45 (6.81%) Confluence
4 37 (5.60%) Elasticsearch
5 20 (3.03%) Apache Pig
6 10 (1.51%) Apache Synapse
6 10 (1.51%) Jupyter
7 5 (0.76%) Apache Solr
7 5 (0.76%) Oracle Coherence
7 5 (0.76%) SAS
8 4 (0.61%) ArcGIS
8 4 (0.61%) Luigi
9 2 (0.30%) Apache Pulsar
9 2 (0.30%) Oracle Workflow
10 1 (0.15%) Camunda
10 1 (0.15%) nginx
10 1 (0.15%) Tornado
Applications
1 5 (0.76%) Stata
2 3 (0.45%) Microsoft Excel
3 2 (0.30%) Chatbot
4 1 (0.15%) Microsoft Office
4 1 (0.15%) Microsoft PowerPoint
4 1 (0.15%) MS Visio
Business Applications
1 2 (0.30%) Temenos T24
2 1 (0.15%) Oracle BPM
2 1 (0.15%) SAP CRM
2 1 (0.15%) SAP ERP
2 1 (0.15%) SAP IS-U
Cloud Services
1 209 (31.62%) AWS
2 139 (21.03%) Azure
3 67 (10.14%) Amazon S3
4 61 (9.23%) GCP
5 54 (8.17%) Amazon EMR
6 37 (5.60%) Serverless
7 34 (5.14%) Azure Data Factory
8 29 (4.39%) Amazon EC2
9 28 (4.24%) Power Platform
10 27 (4.08%) GitHub
11 24 (3.63%) AWS Lambda
12 23 (3.48%) Amazon Kinesis
13 15 (2.27%) Amazon SageMaker
14 13 (1.97%) AWS CloudFormation
14 13 (1.97%) Cloud Computing
15 10 (1.51%) Google Analytics
16 8 (1.21%) Amazon ECS
16 8 (1.21%) Azure Functions
17 7 (1.06%) AWS CodePipeline
17 7 (1.06%) Virtual Private Cloud
Communications & Networking
1 17 (2.57%) HTTP
2 8 (1.21%) Kerberos
3 6 (0.91%) DNS
4 4 (0.61%) Network Security
5 3 (0.45%) Firewall
5 3 (0.45%) VPN
6 2 (0.30%) TCP/IP
7 1 (0.15%) Internet
7 1 (0.15%) Software-Defined Networking
Database & Business Intelligence
1 372 (56.28%) Hadoop
2 335 (50.68%) Big Data
3 205 (31.01%) Apache Hive
4 96 (14.52%) Data Lake
5 95 (14.37%) NoSQL
6 94 (14.22%) Data Warehouse
7 82 (12.41%) HBase
8 71 (10.74%) Impala
9 52 (7.87%) Relational Database
10 45 (6.81%) Spark SQL
11 43 (6.51%) PostgreSQL
12 37 (5.60%) Apache Cassandra
12 37 (5.60%) SQL Server
12 37 (5.60%) Tableau
13 36 (5.45%) Amazon Redshift
13 36 (5.45%) Apache Ignite
14 35 (5.30%) Amazon Athena
15 28 (4.24%) MongoDB
15 28 (4.24%) Power BI
16 27 (4.08%) Parquet
Development Applications
1 91 (13.77%) Jenkins
2 78 (11.80%) Git (software)
3 56 (8.47%) JIRA
4 34 (5.14%) Sonatype Nexus
5 23 (3.48%) Maven
6 21 (3.18%) IBM UrbanCode
7 19 (2.87%) GitLab
8 18 (2.72%) TeamCity
9 15 (2.27%) Gradle
9 15 (2.27%) Selenium
10 14 (2.12%) RStudio
11 12 (1.82%) Cucumber
12 9 (1.36%) Artifactory
12 9 (1.36%) Atlassian Bamboo
12 9 (1.36%) MLflow
13 6 (0.91%) Bitbucket
13 6 (0.91%) JUnit
13 6 (0.91%) SonarQube
14 5 (0.76%) ProC
15 4 (0.61%) IntelliJ
General
1 194 (29.35%) Finance
2 106 (16.04%) Banking
3 84 (12.71%) Front Office
4 48 (7.26%) Investment Banking
5 28 (4.24%) Public Sector
6 22 (3.33%) Marketing
7 17 (2.57%) Retail
8 8 (1.21%) Pharmaceutical
8 8 (1.21%) Retail Banking
9 6 (0.91%) Legal
10 5 (0.76%) Financial Institution
11 4 (0.61%) Games
11 4 (0.61%) Manufacturing
11 4 (0.61%) Telecoms
12 3 (0.45%) Automotive
12 3 (0.45%) French Language
12 3 (0.45%) Publishing
13 2 (0.30%) Electronics
13 2 (0.30%) Military
14 1 (0.15%) Aerospace
Job Titles
1 237 (35.85%) Developer
2 224 (33.89%) Data Engineer
3 69 (10.44%) Scala Developer
4 62 (9.38%) Python Engineer
5 61 (9.23%) Big Data Engineer
6 58 (8.77%) Architect
7 54 (8.17%) AWS Engineer
7 54 (8.17%) Software Engineer
8 52 (7.87%) Java Developer
9 49 (7.41%) Apache Spark Developer
10 48 (7.26%) Data Scientist
11 46 (6.96%) Hadoop Developer
12 44 (6.66%) Big Data Developer
13 38 (5.75%) Python Developer
14 37 (5.60%) Developer/Software Engineer
15 33 (4.99%) Data Architect
16 32 (4.84%) Senior Developer
17 28 (4.24%) Scala Engineer
18 22 (3.33%) Software Developer
19 21 (3.18%) Cloud Engineer
Libraries, Frameworks & Software Standards
1 200 (30.26%) Kafka
2 71 (10.74%) PySpark
2 71 (10.74%) Web Services
3 40 (6.05%) Node.js
3 40 (6.05%) React
4 39 (5.90%) Protocol Buffers
5 36 (5.45%) gRPC
6 35 (5.30%) JSON
7 31 (4.69%) Spark Streaming
8 26 (3.93%) XML
9 23 (3.48%) NumPy
10 21 (3.18%) Spring
11 20 (3.03%) Pandas
12 19 (2.87%) PyTorch
13 18 (2.72%) TensorFlow
14 16 (2.42%) Apache NiFi
14 16 (2.42%) scikit-learn
15 15 (2.27%) RESTful
16 14 (2.12%) Elastic Stack
17 13 (1.97%) HTML
Miscellaneous
1 89 (13.46%) Analytical Skills
2 67 (10.14%) Algorithms
3 45 (6.81%) Data Structures
4 41 (6.20%) Equities
5 37 (5.60%) Fixed Income
6 36 (5.45%) Derivative
7 23 (3.48%) Linux Command Line
8 17 (2.57%) Self-Motivation
9 13 (1.97%) Greenfield Project
10 11 (1.66%) Foreign Exchange (FX)
11 10 (1.51%) Grid Computing
12 9 (1.36%) Cloud Native
12 9 (1.36%) Public Cloud
12 9 (1.36%) User Experience
12 9 (1.36%) Virtual Team
13 8 (1.21%) Analytical Mindset
13 8 (1.21%) Market Risk
14 7 (1.06%) Credit Risk
15 6 (0.91%) Life Science
16 4 (0.61%) PKI
Operating Systems
1 73 (11.04%) Linux
2 18 (2.72%) Unix
3 3 (0.45%) Windows
4 2 (0.30%) Ubuntu
5 1 (0.15%) Android
5 1 (0.15%) CentOS
5 1 (0.15%) Mac OS X
5 1 (0.15%) Red Hat Enterprise Linux
5 1 (0.15%) Windows Server
Processes & Methodologies
1 204 (30.86%) Agile Software Development
2 139 (21.03%) Analytics
3 135 (20.42%) Machine Learning
4 128 (19.36%) Data Science
5 127 (19.21%) DevOps
6 124 (18.76%) ETL
7 96 (14.52%) Data Engineering
8 89 (13.46%) Data Modelling
9 88 (13.31%) Software Engineering
10 83 (12.56%) TDD
11 75 (11.35%) Scrum
12 68 (10.29%) Computer Science
13 64 (9.68%) Business Intelligence
14 61 (9.23%) Mathematics
15 58 (8.77%) BDD
15 58 (8.77%) Visualisation
16 57 (8.62%) Artificial Intelligence
17 54 (8.17%) Test Automation
18 49 (7.41%) Data Ingestion
18 49 (7.41%) Problem-Solving
Programming Languages
1 384 (58.09%) Python
2 268 (40.54%) Scala
3 261 (39.49%) Java
4 256 (38.73%) SQL
5 69 (10.44%) R
6 56 (8.47%) JavaScript
7 44 (6.66%) Kotlin
8 38 (5.75%) C#
9 35 (5.30%) Shell Script
10 27 (4.08%) C++
11 20 (3.03%) Haskell
12 14 (2.12%) Bash Shell
13 13 (1.97%) PL/SQL
14 9 (1.36%) Elm
15 7 (1.06%) Java 8
16 6 (0.91%) T-SQL
16 6 (0.91%) U-SQL
17 5 (0.76%) Groovy
18 4 (0.61%) C
18 4 (0.61%) Go
Qualifications
1 86 (13.01%) Security Cleared
2 74 (11.20%) SC Cleared
3 71 (10.74%) Degree
4 31 (4.69%) Numerate Degree
5 18 (2.72%) Computer Science Degree
6 15 (2.27%) AWS Certification
7 14 (2.12%) TOGAF Certification
8 12 (1.82%) PhD
9 9 (1.36%) Master's Degree
10 5 (0.76%) BPSS Clearance
10 5 (0.76%) DV Cleared
Quality Assurance & Compliance
1 35 (5.30%) QA
2 29 (4.39%) Data Quality
3 3 (0.45%) GDPR
3 3 (0.45%) MiFID
4 2 (0.30%) ISO 20022
4 2 (0.30%) SEPA
4 2 (0.30%) TARGET2
5 1 (0.15%) Basel II
5 1 (0.15%) PCI DSS
5 1 (0.15%) SLA
System Software
1 117 (17.70%) HDFS
2 92 (13.92%) Docker
3 18 (2.72%) Apache ZooKeeper
4 12 (1.82%) Apache Flume
5 3 (0.45%) Active Directory Lightweight Directory Services
5 3 (0.45%) Virtual Machines
6 1 (0.15%) Active Directory
Systems Management
1 98 (14.83%) Kubernetes
2 79 (11.95%) Terraform
3 47 (7.11%) Ansible
4 42 (6.35%) Hadoop YARN
4 42 (6.35%) Oozie
5 13 (1.97%) Puppet
6 8 (1.21%) Opscode Chef
7 7 (1.06%) Grafana
7 7 (1.06%) Kibana
8 6 (0.91%) Prometheus
9 5 (0.76%) Ambari
10 3 (0.45%) Control-M
10 3 (0.45%) Mesos
11 2 (0.30%) Kubeflow
12 1 (0.15%) Chronos
12 1 (0.15%) Knative
12 1 (0.15%) Marathon
12 1 (0.15%) vRealize Orchestrator
Vendors
1 63 (9.53%) Databricks
2 62 (9.38%) Oracle
3 60 (9.08%) Cloudera
4 47 (7.11%) Google
5 36 (5.45%) Bloomberg
6 32 (4.84%) Microsoft
7 29 (4.39%) Informatica
8 24 (3.63%) Hortonworks
9 16 (2.42%) Talend
10 15 (2.27%) Teradata
11 12 (1.82%) Ab Initio
12 11 (1.66%) StreamSets
12 11 (1.66%) TIBCO
13 10 (1.51%) MapR
13 10 (1.51%) SAP
14 8 (1.21%) Sage
14 8 (1.21%) Splunk
15 7 (1.06%) Alteryx
15 7 (1.06%) Confluent
16 6 (0.91%) Atlassian