Period
to 3 August 2021

The following table provides summary statistics for contract job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the contractor rates offered in vacancies that have cited Apache Spark over the 6 months to 3 August 2021 with a comparison to the same period in the previous 2 years.

Note that daily and hourly rates are treated separately in these statistics. When calculating contractor rate percentiles, daily rates are never derived from quoted hourly rates or vice versa.

Apache Spark
UK
6 months to
3 Aug 2021
Same period 2020 Same period 2019
Rank 187 198 165
Rank change year-on-year +11 -33 +30
Contract jobs citing Apache Spark 1,061 487 1,605
As % of all contract jobs advertised in the UK 1.47% 1.35% 1.75%
As % of the Application Platforms category 11.07% 9.14% 11.19%
Number of daily rates quoted 658 310 1,141
10th Percentile £425 £406 £425
25th Percentile £490 £450 £500
Median daily rate (50th Percentile) £575 £525 £550
Median % change year-on-year +9.52% -4.55% -
75th Percentile £654 £637 £625
90th Percentile £713 £688 £700
UK excluding London median daily rate £523 £488 £525
% change year-on-year +7.18% -7.14% +5.00%
Number of hourly rates quoted 6 3 2
10th Percentile £16.38 " £21.59
25th Percentile £19.81 £9.75 £22.10
Median hourly rate £25.75 £10.50 £23.33
Median % change year-on-year +145.24% -54.98% -33.36%
75th Percentile £35.81 £16.88 £24.18
90th Percentile £38.75 £20.25 £24.46
UK excluding London median hourly rate £19.50 £20.00 £23.33
% change year-on-year -2.50% -14.26% +4.83%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all contract job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Contract vacancies with a requirement for application platform skills 9,583 5,326 14,337
As % of all contract IT jobs advertised in the UK 13.24% 14.78% 15.59%
Number of daily rates quoted 6,494 3,562 9,216
10th Percentile £300 £300 £263
25th Percentile £400 £388 £368
Median daily rate (50th Percentile) £500 £475 £473
Median % change year-on-year +5.26% +0.53% +5.00%
75th Percentile £588 £563 £563
90th Percentile £675 £650 £638
UK excluding London median daily rate £450 £450 £425
% change year-on-year - +5.88% +0.59%
Number of hourly rates quoted 180 110 306
10th Percentile £12.25 £12.61 £11.28
25th Percentile £15.44 £15.13 £13.25
Median hourly rate £22.50 £20.00 £20.25
Median % change year-on-year +12.50% -1.23% +1.25%
75th Percentile £52.63 £32.75 £40.07
90th Percentile £68.36 £52.30 £48.75
UK excluding London median hourly rate £20.50 £19.00 £20.00
% change year-on-year +7.89% -5.00% +5.57%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Contractor Daily Rate Trend

3-month moving average daily rate quoted in jobs citing Apache Spark.

Daily rate trend for Apache Spark in the UK

Apache Spark
Daily Rate Histogram

Daily rate distribution for jobs citing Apache Spark over the 6 months to 3 August 2021.

Daily rate histogram for Apache Spark in the UK

Apache Spark
Contractor Hourly Rate Trend

3-month moving average hourly rates quoted in jobs citing Apache Spark.

Hourly rate trend for Apache Spark in the UK

Apache Spark
Hourly Rate Histogram

Hourly rate distribution of jobs citing Apache Spark over the 6 months to 3 August 2021.

Hourly rate histogram for Apache Spark in the UK

Apache Spark
Top 17 Contract Locations

The table below looks at the demand and provides a guide to the median contractor rates quoted in IT jobs citing Apache Spark within the UK over the 6 months to 3 August 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Contract
IT Job Ads
Median
Daily Rate
Past 6 Months
Median Daily Rate
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +19 889 £575 +7.98% 218
London -1 638 £590 +7.27% 147
Work from Home -75 272 £575 +6.88% 33
UK excluding London +8 253 £523 +7.18% 89
South East -15 86 £540 +13.68% 16
North of England 0 65 £565 +18.95% 24
Scotland -49 28 £475 - 12
Yorkshire -8 24 £505 +6.32% 11
North East - 23 £620 - 2
South West +5 22 £504 -4.50% 10
Midlands -27 19 £450 -12.20% 15
North West -33 19 £618 +20.49% 11
Wales -11 18 £485 -3.00% 6
East of England -21 13 £513 -19.61% 4
West Midlands -33 10 £450 -12.20% 13
East Midlands -14 6 £500 - 2
Northern Ireland - 3 £475 - 2

For the 6 months to 3 August 2021, IT contractor jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all contract job ads with a requirement for Apache Spark.

1 542 (51.08%) Python
2 530 (49.95%) Big Data
3 527 (49.67%) Hadoop
4 500 (47.13%) SQL
5 432 (40.72%) Kafka
6 414 (39.02%) Scala
7 390 (36.76%) Java
8 389 (36.66%) Azure
9 346 (32.61%) Apache Hive
10 338 (31.86%) Analytics
11 317 (29.88%) AWS
12 300 (28.28%) Agile Software Development
13 297 (27.99%) DevOps
14 280 (26.39%) Finance
15 253 (23.85%) Apache
16 237 (22.34%) Data Pipeline
17 226 (21.30%) Kubernetes
18 223 (21.02%) Data Engineering
19 217 (20.45%) Social Skills
20 203 (19.13%) Databricks
21 199 (18.76%) NoSQL
22 195 (18.38%) Data Modelling
23 186 (17.53%) Data Warehouse
24 181 (17.06%) Docker
25 176 (16.59%) Jenkins
26 175 (16.49%) ETL
27 174 (16.40%) Data Science
28 172 (16.21%) Git (software)
29 166 (15.65%) GCP
30 161 (15.17%) Banking

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 253 (23.85%) Apache
2 148 (13.95%) Apache Airflow
3 63 (5.94%) Elasticsearch
4 46 (4.34%) Confluence
5 32 (3.02%) Luigi
6 24 (2.26%) Oracle Coherence
7 19 (1.79%) SAS
8 18 (1.70%) Apache Pig
9 13 (1.23%) SharePoint
10 12 (1.13%) Tomcat
11 9 (0.85%) GemFire
12 8 (0.75%) Apache Solr
13 6 (0.57%) Apache Pulsar
13 6 (0.57%) Jupyter
13 6 (0.57%) Oracle Workflow
14 5 (0.47%) Mule
14 5 (0.47%) nginx
15 4 (0.38%) JBoss
16 2 (0.19%) Cloud Foundry
16 2 (0.19%) R Markdown
Applications
1 5 (0.47%) SPSS
2 3 (0.28%) Microsoft PowerPoint
3 1 (0.094%) Chatbot
3 1 (0.094%) Microsoft Excel
3 1 (0.094%) SPSS Modeler
Business Applications
1 8 (0.75%) SAP Oil and Gas
2 2 (0.19%) SAP R/3
2 2 (0.19%) SAP S/4HANA
3 1 (0.094%) Distributed Ledger
3 1 (0.094%) Oracle Cash Management
3 1 (0.094%) Oracle Procure-to-Pay
3 1 (0.094%) Towers Watson Igloo
Cloud Services
1 389 (36.66%) Azure
2 317 (29.88%) AWS
3 166 (15.65%) GCP
4 115 (10.84%) Serverless
5 93 (8.77%) Azure Data Factory
6 87 (8.20%) Amazon S3
6 87 (8.20%) Azure Cosmos DB
7 76 (7.16%) Azure Functions
8 65 (6.13%) Amazon EMR
9 62 (5.84%) GitHub
10 50 (4.71%) Snowflake
11 46 (4.34%) Power Platform
12 45 (4.24%) Azure Synapse Analytics
13 42 (3.96%) Amazon EC2
13 42 (3.96%) AWS Lambda
14 34 (3.20%) Azure Storage
15 30 (2.83%) Collibra
16 28 (2.64%) PaaS
17 24 (2.26%) Cloud Computing
17 24 (2.26%) IaaS
Communications & Networking
1 24 (2.26%) HTTP
2 16 (1.51%) Kerberos
3 14 (1.32%) Firewall
3 14 (1.32%) SSL
4 13 (1.23%) DNS
5 7 (0.66%) HTTPS
6 6 (0.57%) Cisco ASA
7 5 (0.47%) Internet
7 5 (0.47%) SIGTRAN
7 5 (0.47%) SMPP
7 5 (0.47%) SMS
7 5 (0.47%) SS7
7 5 (0.47%) Wireshark
8 4 (0.38%) FTP
8 4 (0.38%) Network Security
9 3 (0.28%) Intranet
9 3 (0.28%) PuTTY
10 1 (0.094%) ADSL
10 1 (0.094%) IPv4
10 1 (0.094%) IPv6
Database & Business Intelligence
1 530 (49.95%) Big Data
2 527 (49.67%) Hadoop
3 346 (32.61%) Apache Hive
4 199 (18.76%) NoSQL
5 186 (17.53%) Data Warehouse
6 149 (14.04%) HBase
7 143 (13.48%) Data Lake
8 130 (12.25%) Relational Database
9 119 (11.22%) MongoDB
10 98 (9.24%) PostgreSQL
11 82 (7.73%) Impala
12 79 (7.45%) Metadata
13 77 (7.26%) BigQuery
14 75 (7.07%) Apache Cassandra
15 73 (6.88%) SQL Server
16 70 (6.60%) Spark SQL
17 62 (5.84%) Tableau
18 58 (5.47%) Amazon Redshift
19 54 (5.09%) Azure SQL Database
20 53 (5.00%) Parquet
Development Applications
1 176 (16.59%) Jenkins
2 172 (16.21%) Git (software)
3 70 (6.60%) JIRA
4 44 (4.15%) Maven
5 35 (3.30%) GitLab
6 32 (3.02%) Sonatype Nexus
7 30 (2.83%) TeamCity
8 24 (2.26%) Artifactory
8 24 (2.26%) Hudson
9 22 (2.07%) ERwin
9 22 (2.07%) PowerDesigner
9 22 (2.07%) Subversion
10 20 (1.89%) IBM UrbanCode
11 19 (1.79%) Bitbucket
12 16 (1.51%) IntelliJ
13 13 (1.23%) Oracle APEX
13 13 (1.23%) Visual Studio Team System
14 12 (1.13%) JUnit
15 11 (1.04%) Visual Studio
16 10 (0.94%) Cucumber
General
1 280 (26.39%) Finance
2 217 (20.45%) Social Skills
3 161 (15.17%) Banking
4 135 (12.72%) Analytical Skills
5 46 (4.34%) Investment Banking
6 36 (3.39%) Front Office
7 35 (3.30%) Presentation Skills
7 35 (3.30%) Public Sector
8 25 (2.36%) Marketing
9 14 (1.32%) Financial Institution
9 14 (1.32%) Retail Banking
10 11 (1.04%) Documentation Skills
10 11 (1.04%) Publishing
11 9 (0.85%) Retail
12 7 (0.66%) Games
13 6 (0.57%) Back Office
14 5 (0.47%) French Language
14 5 (0.47%) Local Government
15 4 (0.38%) Influencing Skills
15 4 (0.38%) Manufacturing
Job Titles
1 406 (38.27%) Data Engineer
2 241 (22.71%) Developer
3 108 (10.18%) Architect
4 83 (7.82%) Big Data Engineer
5 82 (7.73%) Azure Engineer
6 80 (7.54%) Java Developer
7 73 (6.88%) Data Architect
7 73 (6.88%) Scala Developer
8 63 (5.94%) AWS Engineer
9 59 (5.56%) Data Scientist
10 54 (5.09%) DevOps Engineer
11 43 (4.05%) Apache Spark Developer
11 43 (4.05%) Scala Engineer
12 42 (3.96%) Senior Developer
13 39 (3.68%) Senior Data Engineer
14 35 (3.30%) Big Data Developer
15 31 (2.92%) Hadoop Developer
15 31 (2.92%) Java Engineer
15 31 (2.92%) Python Engineer
16 30 (2.83%) Cloud Engineer
Libraries, Frameworks & Software Standards
1 432 (40.72%) Kafka
2 129 (12.16%) PySpark
3 101 (9.52%) Spark Streaming
4 68 (6.41%) Spring
5 49 (4.62%) JSON
5 49 (4.62%) Web Services
6 48 (4.52%) RESTful
7 39 (3.68%) Apache Avro
8 32 (3.02%) Spring Boot
9 30 (2.83%) Apache NiFi
9 30 (2.83%) Node.js
10 27 (2.54%) ARM Templates
10 27 (2.54%) Pandas
11 25 (2.36%) .NET
11 25 (2.36%) Akka
12 23 (2.17%) REST
13 21 (1.98%) NumPy
14 19 (1.79%) AngularJS
14 19 (1.79%) TensorFlow
15 17 (1.60%) Samza
Miscellaneous
1 71 (6.69%) Data Structures
2 64 (6.03%) Algorithms
3 43 (4.05%) Virtual Team
4 39 (3.68%) Market Risk
5 29 (2.73%) Greenfield Project
6 28 (2.64%) Grid Computing
7 26 (2.45%) iPad
8 24 (2.26%) Cloud Native
8 24 (2.26%) Management Information System
9 23 (2.17%) Linux Command Line
10 19 (1.79%) Public Cloud
11 17 (1.60%) Data Protection Act
12 16 (1.51%) Legacy Applications
13 15 (1.41%) Data Centre
14 14 (1.32%) Legacy Systems
14 14 (1.32%) Self-Motivation
15 13 (1.23%) PKI
16 12 (1.13%) Replication
17 10 (0.94%) Hedge funds
17 10 (0.94%) Social Media
Operating Systems
1 120 (11.31%) Linux
2 83 (7.82%) Unix
3 23 (2.17%) Windows
4 15 (1.41%) Red Hat Enterprise Linux
5 6 (0.57%) Android
6 3 (0.28%) CentOS
6 3 (0.28%) Debian
6 3 (0.28%) Ubuntu
6 3 (0.28%) VMS
7 1 (0.094%) Apple iOS
Processes & Methodologies
1 338 (31.86%) Analytics
2 300 (28.28%) Agile Software Development
3 297 (27.99%) DevOps
4 237 (22.34%) Data Pipeline
5 223 (21.02%) Data Engineering
6 195 (18.38%) Data Modelling
7 175 (16.49%) ETL
8 174 (16.40%) Data Science
9 136 (12.82%) Machine Learning
10 128 (12.06%) Business Intelligence
11 125 (11.78%) Scrum
12 118 (11.12%) Data Architecture
13 106 (9.99%) Microservices
14 103 (9.71%) Computer Science
15 89 (8.39%) TDD
16 88 (8.29%) Migration
17 86 (8.11%) Data Management
17 86 (8.11%) Open Source
17 86 (8.11%) Software Engineering
18 83 (7.82%) Data Ingestion
Programming Languages
1 542 (51.08%) Python
2 500 (47.13%) SQL
3 414 (39.02%) Scala
4 390 (36.76%) Java
5 78 (7.35%) R
6 43 (4.05%) C#
7 38 (3.58%) JavaScript
8 30 (2.83%) PL/SQL
8 30 (2.83%) T-SQL
9 29 (2.73%) Java 8
9 29 (2.73%) Shell Script
10 27 (2.54%) C++
10 27 (2.54%) Groovy
11 23 (2.17%) Bash Shell
12 21 (1.98%) Go
13 18 (1.70%) Kotlin
14 17 (1.60%) PowerShell
15 13 (1.23%) Perl
16 12 (1.13%) C
16 12 (1.13%) Haskell
Qualifications
1 97 (9.14%) Degree
2 75 (7.07%) Security Cleared
3 54 (5.09%) SC Cleared
4 44 (4.15%) Computer Science Degree
5 16 (1.51%) Numerate Degree
6 11 (1.04%) BPSS Clearance
6 11 (1.04%) ISEB
7 10 (0.94%) PhD
8 7 (0.66%) AWS Certification
9 6 (0.57%) Master's Degree
10 3 (0.28%) DBS Check
11 2 (0.19%) TOGAF Certification
12 1 (0.094%) CBAP
12 1 (0.094%) CCA
12 1 (0.094%) PMI Certification
Quality Assurance & Compliance
1 68 (6.41%) Data Quality
2 40 (3.77%) QA
3 30 (2.83%) GDPR
4 8 (0.75%) IFRS
5 7 (0.66%) Disclosure Scotland
6 3 (0.28%) MiFID
6 3 (0.28%) SLA
7 2 (0.19%) COBIT
7 2 (0.19%) ISO/IEC 27001
7 2 (0.19%) NIST
7 2 (0.19%) Sarbanes-Oxley
8 1 (0.094%) Dodd-Frank
8 1 (0.094%) EMIR
8 1 (0.094%) ISO 20022
8 1 (0.094%) PSD2
8 1 (0.094%) TARGET2
System Software
1 181 (17.06%) Docker
2 142 (13.38%) HDFS
3 26 (2.45%) Apache ZooKeeper
4 16 (1.51%) Apache Flume
5 14 (1.32%) Active Directory
6 5 (0.47%) Virtual Machines
7 1 (0.094%) VMware Infrastructure
Systems Management
1 226 (21.30%) Kubernetes
2 136 (12.82%) Terraform
3 78 (7.35%) Oozie
4 67 (6.31%) Ansible
5 49 (4.62%) Hadoop YARN
6 26 (2.45%) Prometheus
7 25 (2.36%) Puppet
8 21 (1.98%) Grafana
9 18 (1.70%) Ambari
10 14 (1.32%) Kibana
10 14 (1.32%) Opscode Chef
11 12 (1.13%) Control-M
12 11 (1.04%) Single Sign-On
13 9 (0.85%) logstash
14 7 (0.66%) Active Directory Federation Services
14 7 (0.66%) Docker Swarm
15 6 (0.57%) FortiGate
15 6 (0.57%) Rancher
16 4 (0.38%) Istio
17 3 (0.28%) Apache Slider
Vendors
1 203 (19.13%) Databricks
2 119 (11.22%) Oracle
3 110 (10.37%) Cloudera
4 96 (9.05%) Google
5 77 (7.26%) Microsoft
6 54 (5.09%) Informatica
7 31 (2.92%) Confluent
7 31 (2.92%) Hortonworks
8 29 (2.73%) BiZZdesign
9 28 (2.64%) SAP
10 25 (2.36%) IBM
11 20 (1.89%) Splunk
11 20 (1.89%) Teradata
12 19 (1.79%) Alteryx
13 18 (1.70%) StreamSets
14 15 (1.41%) Ab Initio
15 13 (1.23%) MapR
15 13 (1.23%) Red Hat
15 13 (1.23%) Salesforce.com
16 9 (0.85%) Talend