Period
to 27 November 2020

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 27 November 2020 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
27 Nov 2020
Same period 2019 Same period 2018
Rank 192 205 206
Rank change year-on-year +13 +1 -18
Contract jobs citing Apache Spark 480 1,082 1,429
As % of all contract jobs advertised in the UK 1.30% 1.31% 1.37%
As % of the Application Platforms category 8.85% 8.94% 8.17%
Number of daily rates quoted 368 781 1,000
Median daily rate £550 £575 £550
Median daily rate % change year-on-year -4.35% +4.55% -
10th Percentile £384 £425 £425
90th Percentile £704 £725 £713
UK excluding London median daily rate £428 £525 £525
% change year-on-year -18.57% - +10.53%
Number of hourly rates quoted 14 4 6
Median hourly rate £30.02 £29.83 £33.75
Median hourly rate % change year-on-year +0.66% -11.63% +44.11%
10th Percentile £11.55 £22.27 £18.13
90th Percentile £35.00 £48.13 £112.13
UK excluding London median hourly rate £35.00 £24.65 £22.25
% change year-on-year +41.99% +10.79% -5.00%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 5,421 12,097 17,491
As % of all contract IT jobs advertised in the UK 14.73% 14.68% 16.81%
Number of daily rates quoted 3,333 7,759 11,324
Median daily rate £465 £450 £450
Median daily rate % change year-on-year +3.33% - -
10th Percentile £275 £250 £275
90th Percentile £650 £633 £625
UK excluding London median daily rate £430 £425 £425
% change year-on-year +1.18% - +6.25%
Number of hourly rates quoted 217 327 404
Median hourly rate £35.00 £21.50 £21.50
Median hourly rate % change year-on-year +62.79% - +7.50%
10th Percentile £13.43 £11.22 £12.19
90th Percentile £60.55 £53.00 £47.50
Median hourly rate £25.40 £20.50 £20.75
% change year-on-year +23.90% -1.20% +3.75%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 27 November 2020.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 27 November 2020.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 16 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 27 November 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +2 415 £561 -2.39% 112
London +8 333 £575 - 71
Work from Home -82 88 £550 -17.60% 32
UK excluding London +12 85 £428 -18.57% 50
South East +87 30 £425 -19.05% 12
North of England +58 19 £401 -27.09% 19
East of England +64 16 £438 -18.60% 5
North West +54 14 £345 -37.27% 5
Scotland -2 8 £410 -21.90% 5
South West +84 6 £575 -4.17% 1
Wales +34 5 £439 +11.43% 4
Yorkshire +33 5 £513 -6.82% 14
Midlands +47 3 £400 -23.81% 4
East Midlands +38 2 - - 3
Northern Ireland +16 2 £450 -
West Midlands +45 1 £400 -23.81% 1

For the 6 months to 27 November 2020, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 304 (63.33%) Python
2 269 (56.04%) Hadoop
3 266 (55.42%) Big Data
4 186 (38.75%) Java
5 184 (38.33%) Scala
6 180 (37.50%) SQL
7 164 (34.17%) AWS
8 138 (28.75%) Apache Hive
9 130 (27.08%) Kafka
10 126 (26.25%) Finance
11 118 (24.58%) Apache
11 118 (24.58%) Azure
12 111 (23.13%) ETL
12 111 (23.13%) Agile Software Development
13 100 (20.83%) Analytics
14 98 (20.42%) Data Science
15 92 (19.17%) Kubernetes
16 88 (18.33%) Machine Learning
17 84 (17.50%) Docker
18 80 (16.67%) DevOps
19 77 (16.04%) Security Cleared
19 77 (16.04%) Data Modelling
20 76 (15.83%) Banking
21 71 (14.79%) Front Office
22 69 (14.38%) Data Engineering
23 68 (14.17%) HDFS
24 66 (13.75%) NoSQL
25 64 (13.33%) Visualisation
26 62 (12.92%) Software Engineering
26 62 (12.92%) SC Cleared

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 118 (24.58%) Apache
2 29 (6.04%) Elasticsearch
3 28 (5.83%) Apache Airflow
4 24 (5.00%) Apache Pig
5 18 (3.75%) SAS
6 15 (3.13%) Confluence
7 7 (1.46%) Jupyter
8 4 (0.83%) ArcGIS
8 4 (0.83%) Tornado
9 3 (0.63%) Apache Synapse
9 3 (0.63%) Luigi
10 2 (0.42%) Oracle Coherence
11 1 (0.21%) Apache Solr
11 1 (0.21%) Camunda
11 1 (0.21%) nginx
11 1 (0.21%) Oracle Workflow
11 1 (0.21%) R Markdown
Applications
1 3 (0.63%) Microsoft Excel
2 2 (0.42%) Chatbot
3 1 (0.21%) Microsoft PowerPoint
Business Applications
1 4 (0.83%) SAP Oil and Gas
2 1 (0.21%) Oracle BPM
2 1 (0.21%) Sparx Enterprise Architect
Cloud Services
1 164 (34.17%) AWS
2 118 (24.58%) Azure
3 58 (12.08%) GCP
4 38 (7.92%) Amazon EMR
5 37 (7.71%) Amazon S3
6 25 (5.21%) Serverless
7 24 (5.00%) Azure Data Factory
8 16 (3.33%) Amazon EC2
8 16 (3.33%) Power Platform
9 12 (2.50%) Virtual Private Cloud
10 11 (2.29%) AWS Lambda
11 10 (2.08%) Amazon SageMaker
12 9 (1.88%) Amazon Kinesis
12 9 (1.88%) GitHub
13 8 (1.67%) Amazon EKS
14 7 (1.46%) Azure Functions
14 7 (1.46%) Cloud Computing
15 5 (1.04%) Azure Active Directory
15 5 (1.04%) SaaS
16 4 (0.83%) AWS CloudFormation
Communications & Networking
1 7 (1.46%) Kerberos
2 6 (1.25%) SSL
3 4 (0.83%) VPN
4 3 (0.63%) HTTP
5 2 (0.42%) Firewall
5 2 (0.42%) TCP/IP
6 1 (0.21%) HTTPS
6 1 (0.21%) Internet
6 1 (0.21%) Skype
6 1 (0.21%) Wi-Fi
6 1 (0.21%) Wireless
Database & Business Intelligence
1 269 (56.04%) Hadoop
2 266 (55.42%) Big Data
3 138 (28.75%) Apache Hive
4 66 (13.75%) NoSQL
5 59 (12.29%) Impala
6 46 (9.58%) Data Warehouse
7 38 (7.92%) Data Lake
8 35 (7.29%) Apache Ignite
9 34 (7.08%) Relational Database
10 32 (6.67%) HBase
10 32 (6.67%) Tableau
11 25 (5.21%) PostgreSQL
12 22 (4.58%) Apache Cassandra
13 21 (4.38%) Data Mining
14 19 (3.96%) Amazon RDS
15 18 (3.75%) Spark SQL
15 18 (3.75%) SQL Server
16 17 (3.54%) BigQuery
17 16 (3.33%) Amazon Redshift
17 16 (3.33%) MongoDB
Development Applications
1 60 (12.50%) Git (software)
2 52 (10.83%) Jenkins
3 20 (4.17%) JIRA
4 14 (2.92%) Selenium
4 14 (2.92%) TeamCity
5 11 (2.29%) GitLab
6 8 (1.67%) Cucumber
7 6 (1.25%) kappa
8 5 (1.04%) Artifactory
8 5 (1.04%) Atlassian Bamboo
8 5 (1.04%) ProC
9 4 (0.83%) Hudson
9 4 (0.83%) Maven
9 4 (0.83%) MLflow
10 3 (0.63%) Gradle
10 3 (0.63%) SonarQube
11 2 (0.42%) ERwin
12 1 (0.21%) Bitbucket
12 1 (0.21%) JUnit
12 1 (0.21%) RStudio
General
1 126 (26.25%) Finance
2 76 (15.83%) Banking
3 71 (14.79%) Front Office
4 24 (5.00%) Investment Banking
5 20 (4.17%) Public Sector
6 16 (3.33%) Marketing
6 16 (3.33%) Retail
7 9 (1.88%) Pharmaceutical
8 8 (1.67%) Manufacturing
9 7 (1.46%) Electronics
9 7 (1.46%) Retail Banking
10 6 (1.25%) Telecoms
11 4 (0.83%) Aerospace
11 4 (0.83%) Games
11 4 (0.83%) Legal
12 3 (0.63%) Financial Institution
13 2 (0.42%) Law
13 2 (0.42%) Military
13 2 (0.42%) Publishing
14 1 (0.21%) Automotive
Job Titles
1 174 (36.25%) Data Engineer
2 153 (31.88%) Developer
3 58 (12.08%) Python Engineer
4 54 (11.25%) Big Data Engineer
5 42 (8.75%) Software Engineer
6 38 (7.92%) Architect
7 35 (7.29%) Data Scientist
7 35 (7.29%) Developer/Software Engineer
8 34 (7.08%) Python Developer
9 33 (6.88%) Java Developer
10 31 (6.46%) Scala Developer
11 26 (5.42%) AWS Engineer
11 26 (5.42%) Azure Engineer
12 25 (5.21%) Senior Data Engineer
12 25 (5.21%) Senior Developer
13 23 (4.79%) Data Architect
14 20 (4.17%) Hadoop Engineer
15 18 (3.75%) Big Data Developer
15 18 (3.75%) Python Software Engineer
15 18 (3.75%) Software Developer
Libraries, Frameworks & Software Standards
1 130 (27.08%) Kafka
2 55 (11.46%) PySpark
3 49 (10.21%) Web Services
4 39 (8.13%) React
5 38 (7.92%) Node.js
6 35 (7.29%) gRPC
6 35 (7.29%) Protocol Buffers
7 28 (5.83%) Spark Streaming
8 23 (4.79%) JSON
9 20 (4.17%) TensorFlow
10 18 (3.75%) Akka
10 18 (3.75%) scikit-learn
11 16 (3.33%) Pandas
12 15 (3.13%) Elastic Stack
12 15 (3.13%) PyTorch
12 15 (3.13%) Spring
13 12 (2.50%) NumPy
13 12 (2.50%) XML
14 10 (2.08%) .NET
14 10 (2.08%) REST
Miscellaneous
1 55 (11.46%) Analytical Skills
2 45 (9.38%) Algorithms
3 38 (7.92%) Fixed Income
4 37 (7.71%) Equities
5 35 (7.29%) Derivative
6 17 (3.54%) Data Structures
7 15 (3.13%) Greenfield Project
8 13 (2.71%) Analytical Mindset
8 13 (2.71%) Foreign Exchange (FX)
9 11 (2.29%) Credit Risk
9 11 (2.29%) FMCG
10 10 (2.08%) Cloud Native
11 8 (1.67%) Public Cloud
12 6 (1.25%) Life Science
12 6 (1.25%) Market Risk
12 6 (1.25%) Reference Data
12 6 (1.25%) Virtual Team
13 5 (1.04%) Management Information System
13 5 (1.04%) Self-Motivation
13 5 (1.04%) User Experience
Operating Systems
1 38 (7.92%) Linux
2 9 (1.88%) Unix
3 6 (1.25%) Red Hat Enterprise Linux
3 6 (1.25%) Ubuntu
4 4 (0.83%) Debian
4 4 (0.83%) Windows
5 2 (0.42%) CentOS
Processes & Methodologies
1 111 (23.13%) Agile Software Development
1 111 (23.13%) ETL
2 100 (20.83%) Analytics
3 98 (20.42%) Data Science
4 88 (18.33%) Machine Learning
5 80 (16.67%) DevOps
6 77 (16.04%) Data Modelling
7 69 (14.38%) Data Engineering
8 64 (13.33%) Visualisation
9 62 (12.92%) Software Engineering
10 51 (10.63%) Computer Science
11 48 (10.00%) Business Intelligence
11 48 (10.00%) ETL Development
11 48 (10.00%) TDD
12 46 (9.58%) Microservices
13 44 (9.17%) Data Visualisation
13 44 (9.17%) Mathematics
14 40 (8.33%) Asset Management
15 38 (7.92%) Investment Management
16 36 (7.50%) Scrum
Programming Languages
1 304 (63.33%) Python
2 186 (38.75%) Java
3 184 (38.33%) Scala
4 180 (37.50%) SQL
5 55 (11.46%) JavaScript
6 49 (10.21%) R
7 46 (9.58%) Kotlin
8 43 (8.96%) C#
9 23 (4.79%) C++
10 17 (3.54%) Shell Script
11 14 (2.92%) Haskell
12 12 (2.50%) Bash Shell
13 10 (2.08%) Perl
14 8 (1.67%) Java 8
15 6 (1.25%) PHP
15 6 (1.25%) T-SQL
16 5 (1.04%) Ruby
17 4 (0.83%) Go
17 4 (0.83%) Groovy
17 4 (0.83%) PowerShell
Qualifications
1 77 (16.04%) Security Cleared
2 62 (12.92%) Degree
2 62 (12.92%) SC Cleared
3 30 (6.25%) Numerate Degree
4 18 (3.75%) Computer Science Degree
5 13 (2.71%) TOGAF Certification
6 11 (2.29%) Master's Degree
7 9 (1.88%) AWS Certification
8 8 (1.67%) DV Cleared
9 7 (1.46%) PhD
10 1 (0.21%) CSPO
10 1 (0.21%) Postgraduate
Quality Assurance & Compliance
1 14 (2.92%) QA
2 12 (2.50%) Data Quality
3 7 (1.46%) MiFID
4 2 (0.42%) Sarbanes-Oxley
4 2 (0.42%) SLA
5 1 (0.21%) CDISC
5 1 (0.21%) GDPR
5 1 (0.21%) GxP
5 1 (0.21%) ISO 20022
5 1 (0.21%) PCI DSS
5 1 (0.21%) SEPA
5 1 (0.21%) TARGET2
System Software
1 84 (17.50%) Docker
2 68 (14.17%) HDFS
3 10 (2.08%) Apache ZooKeeper
4 8 (1.67%) Apache Flume
5 5 (1.04%) Active Directory
6 4 (0.83%) Active Directory Lightweight Directory Services
7 1 (0.21%) Virtual Machines
Systems Management
1 92 (19.17%) Kubernetes
2 61 (12.71%) Terraform
3 30 (6.25%) Ansible
4 21 (4.38%) Hadoop YARN
5 12 (2.50%) Oozie
6 9 (1.88%) Puppet
7 5 (1.04%) Opscode Chef
8 2 (0.42%) Control-M
8 2 (0.42%) Fluentd
8 2 (0.42%) Knative
8 2 (0.42%) Kubeflow
8 2 (0.42%) logstash
8 2 (0.42%) Mesos
8 2 (0.42%) Prometheus
9 1 (0.21%) Ambari
9 1 (0.21%) Grafana
9 1 (0.21%) Istio
9 1 (0.21%) Kibana
9 1 (0.21%) vRealize Orchestrator
9 1 (0.21%) Zipkin
Vendors
1 56 (11.67%) Cloudera
2 54 (11.25%) Databricks
3 35 (7.29%) Bloomberg
3 35 (7.29%) Oracle
4 33 (6.88%) Google
5 23 (4.79%) Informatica
6 20 (4.17%) Hortonworks
7 17 (3.54%) Microsoft
8 16 (3.33%) Teradata
9 13 (2.71%) Ab Initio
10 10 (2.08%) Red Hat
11 9 (1.88%) SAP
11 9 (1.88%) StreamSets
12 8 (1.67%) Sage
12 8 (1.67%) TIBCO
13 7 (1.46%) Alteryx
13 7 (1.46%) MuleSoft
13 7 (1.46%) Splunk
13 7 (1.46%) Talend
14 6 (1.25%) Atlassian