Period
to 25 May 2020

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 25 May 2020 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
25 May 2020
Same period 2019 Same period 2018
Rank 244 230 231
Rank change year-on-year -14 +1 +72
Permanent jobs citing Apache Spark 1,241 2,169 2,670
As % of all permanent jobs advertised in the UK 1.35% 1.49% 1.54%
As % of the Application Platforms category 9.37% 9.73% 9.14%
Number of salaries quoted 965 1,804 2,185
Median annual salary £75,000 £75,000 £70,000
Median salary % change year-on-year - +7.14% +3.70%
10th Percentile £47,500 £47,500 £43,750
90th Percentile £105,000 £108,750 £102,500
UK excluding London median annual salary £62,500 £60,000 £60,000
% change year-on-year +4.17% - +5.26%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 13,246 22,291 29,222
As % of all permanent IT jobs advertised in the UK 14.36% 15.36% 16.80%
Number of salaries quoted 10,495 17,455 23,692
Median annual salary £52,500 £50,000 £50,000
Median salary % change year-on-year +5.00% - +8.70%
10th Percentile £30,000 £28,000 £28,000
90th Percentile £87,500 £85,000 £80,000
UK excluding London median annual salary £45,000 £43,000 £42,500
% change year-on-year +4.65% +1.18% +6.25%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a proportion of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

3-month moving average salary quoted in jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

Salary distribution for jobs citing Apache Spark over the 6 months to 25 May 2020.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 18 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 25 May 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -10 1,175 £75,000 - 117
London -31 844 £77,500 -3.13% 74
UK excluding London +36 395 £62,500 +4.17% 53
South East +104 139 £67,500 -10.00% 17
Work from Home +6 74 £75,000 - 16
Midlands +78 72 £72,500 +11.54%
North of England +38 58 £55,000 -4.35% 12
Scotland +51 50 £55,000 +4.76% 6
East Midlands +25 46 £75,000 +30.43%
East of England +69 35 £75,000 +15.38% 12
North West +40 35 £55,000 -15.38% 7
South West -34 27 £50,000 -9.09% 2
West Midlands +74 26 £55,000 -21.43%
Yorkshire +70 19 £52,500 +6.41% 5
Wales +17 10 £50,000 +55.04% 4
North East +5 4 £56,250 +14.01%
Northern Ireland +18 3 £55,000 +29.41%
Channel Islands -3 1 - -

For the 6 months to 25 May 2020, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 823 (66.32%) Hadoop
2 817 (65.83%) Python
3 767 (61.80%) Big Data
4 557 (44.88%) Java
5 552 (44.48%) AWS
6 509 (41.02%) Analytics
7 506 (40.77%) SQL
8 500 (40.29%) Agile Software Development
9 485 (39.08%) Kafka
10 478 (38.52%) Scala
11 451 (36.34%) Data Science
12 381 (30.70%) Machine Learning
13 350 (28.20%) Azure
14 329 (26.51%) NoSQL
15 302 (24.34%) Finance
16 288 (23.21%) GCP
17 274 (22.08%) DevOps
18 267 (21.51%) Docker
19 258 (20.79%) ETL
20 255 (20.55%) Apache Hive
21 253 (20.39%) Degree
22 242 (19.50%) Data Engineering
23 225 (18.13%) Kubernetes
24 224 (18.05%) Software Engineering
25 210 (16.92%) Artificial Intelligence
26 206 (16.60%) Apache
27 202 (16.28%) Data Warehouse
28 201 (16.20%) R
29 194 (15.63%) Data Modelling
30 192 (15.47%) Business Intelligence

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 206 (16.60%) Apache
2 103 (8.30%) Elasticsearch
3 71 (5.72%) Apache Airflow
4 41 (3.30%) Apache Solr
5 38 (3.06%) Jupyter
6 31 (2.50%) SAS
7 29 (2.34%) Luigi
8 19 (1.53%) Confluence
9 15 (1.21%) Apache Pig
10 11 (0.89%) CMS
10 11 (0.89%) Tomcat
11 10 (0.81%) Mule
12 8 (0.64%) Oracle Coherence
13 7 (0.56%) SharePoint
14 6 (0.48%) JBoss
14 6 (0.48%) Tornado
14 6 (0.48%) WildFly
15 5 (0.40%) Camunda
15 5 (0.40%) Hyperledger
15 5 (0.40%) nginx
Applications
1 8 (0.64%) SPSS
2 6 (0.48%) Microsoft Excel
3 3 (0.24%) Gephi
4 2 (0.16%) Microsoft PowerPoint
4 2 (0.16%) MS Visio
5 1 (0.081%) Chatbot
5 1 (0.081%) Weka
Business Applications
1 5 (0.40%) SAP CRM
1 5 (0.40%) SAP ERP
1 5 (0.40%) SAP IS-U
2 3 (0.24%) Sparx Enterprise Architect
3 2 (0.16%) SAP BPC
3 2 (0.16%) SAP S/4HANA
4 1 (0.081%) Adobe Campaign
Cloud Services
1 552 (44.48%) AWS
2 350 (28.20%) Azure
3 288 (23.21%) GCP
4 117 (9.43%) Amazon EMR
5 99 (7.98%) Power Platform
6 96 (7.74%) Amazon S3
7 66 (5.32%) Amazon Kinesis
8 58 (4.67%) Serverless
9 49 (3.95%) Amazon EC2
10 41 (3.30%) Azure Data Factory
11 34 (2.74%) Cloud Computing
12 30 (2.42%) AWS Lambda
13 26 (2.10%) IBM Cloud
14 24 (1.93%) SaaS
15 22 (1.77%) PaaS
16 20 (1.61%) Amazon CloudWatch
16 20 (1.61%) Amazon SQS
17 17 (1.37%) Dropbox
17 17 (1.37%) Slack
18 16 (1.29%) GitHub
Communications & Networking
1 13 (1.05%) FTP
2 12 (0.97%) HTTP
3 8 (0.64%) Intrusion Detection
3 8 (0.64%) SSL
4 5 (0.40%) TCP/IP
5 4 (0.32%) Firewall
6 3 (0.24%) 5G
6 3 (0.24%) DNS
6 3 (0.24%) SAN
7 2 (0.16%) 3G
7 2 (0.16%) 4G
7 2 (0.16%) AMQP
7 2 (0.16%) Ethernet
7 2 (0.16%) HTTPS
7 2 (0.16%) LTE
7 2 (0.16%) MPLS
7 2 (0.16%) VPN
7 2 (0.16%) Wi-Fi
7 2 (0.16%) WLAN
8 1 (0.081%) Network Security
Database & Business Intelligence
1 823 (66.32%) Hadoop
2 767 (61.80%) Big Data
3 329 (26.51%) NoSQL
4 255 (20.55%) Apache Hive
5 202 (16.28%) Data Warehouse
6 178 (14.34%) Apache Cassandra
7 154 (12.41%) Amazon Redshift
8 151 (12.17%) MongoDB
9 136 (10.96%) SQL Server
10 131 (10.56%) HBase
11 128 (10.31%) PostgreSQL
12 124 (9.99%) Data Lake
13 109 (8.78%) Relational Database
14 99 (7.98%) Impala
14 99 (7.98%) Power BI
15 96 (7.74%) Neo4j
16 77 (6.20%) Flink
17 67 (5.40%) BigQuery
18 57 (4.59%) Amazon Athena
19 47 (3.79%) Amazon DynamoDB
Development Applications
1 144 (11.60%) Git (software)
2 120 (9.67%) Jenkins
3 57 (4.59%) Atlassian Bamboo
4 48 (3.87%) JIRA
5 38 (3.06%) JUnit
6 36 (2.90%) Vagrant
7 24 (1.93%) Maven
8 21 (1.69%) TeamCity
9 16 (1.29%) Visual Studio
10 14 (1.13%) Bitbucket
11 13 (1.05%) Sonatype Nexus
11 13 (1.05%) Visual Studio Team System
12 12 (0.97%) GitLab
13 9 (0.73%) Artifactory
13 9 (0.73%) Mockito
14 8 (0.64%) CircleCI
14 8 (0.64%) Cucumber
14 8 (0.64%) git-flow
14 8 (0.64%) Octopus Deploy
15 7 (0.56%) GoCD
General
1 302 (24.34%) Finance
2 102 (8.22%) Banking
3 74 (5.96%) Retail
4 59 (4.75%) Marketing
5 44 (3.55%) Front Office
6 36 (2.90%) Manufacturing
7 35 (2.82%) Telecoms
8 31 (2.50%) Public Sector
9 30 (2.42%) Investment Banking
10 25 (2.01%) Games
11 21 (1.69%) Automotive
11 21 (1.69%) Legal
12 14 (1.13%) Aerospace
13 9 (0.73%) Advertising
14 5 (0.40%) Financial Institution
14 5 (0.40%) Retail Banking
15 4 (0.32%) Billing
16 3 (0.24%) Electronics
16 3 (0.24%) Pharmaceutical
17 2 (0.16%) Back Office
Job Titles
1 420 (33.84%) Data Engineer
2 256 (20.63%) Developer
3 155 (12.49%) Architect
4 128 (10.31%) Java Developer
5 119 (9.59%) Big Data Engineer
5 119 (9.59%) Data Architect
6 104 (8.38%) Data Scientist
7 79 (6.37%) Senior Developer
8 68 (5.48%) Senior Data Engineer
9 62 (5.00%) Python Engineer
10 59 (4.75%) Software Engineer
11 53 (4.27%) AWS Engineer
11 53 (4.27%) Big Data Architect
12 51 (4.11%) Senior Data Scientist
13 47 (3.79%) Senior Java Developer
14 45 (3.63%) Scala Developer
15 34 (2.74%) Cloud Engineer
16 32 (2.58%) Analyst
16 32 (2.58%) DevOps Engineer
17 30 (2.42%) Platform Engineer
Libraries, Frameworks & Software Standards
1 485 (39.08%) Kafka
2 95 (7.66%) Spring
3 86 (6.93%) Natural Language Processing
4 81 (6.53%) Spark Streaming
5 79 (6.37%) TensorFlow
6 68 (5.48%) Node.js
6 68 (5.48%) React
7 64 (5.16%) scikit-learn
8 61 (4.92%) Spring Boot
9 58 (4.67%) Web Services
10 53 (4.27%) AngularJS
10 53 (4.27%) REST
11 48 (3.87%) HTML
12 45 (3.63%) .NET
12 45 (3.63%) PySpark
13 43 (3.46%) Pandas
14 42 (3.38%) NumPy
15 35 (2.82%) RabbitMQ
16 31 (2.50%) Apache NiFi
17 30 (2.42%) PyTorch
Miscellaneous
1 165 (13.30%) Analytical Skills
2 123 (9.91%) Algorithms
3 76 (6.12%) Greenfield Project
4 59 (4.75%) Data Structures
5 45 (3.63%) Public Cloud
6 43 (3.46%) Blog
6 43 (3.46%) Hedge funds
7 36 (2.90%) Cloud Native
8 32 (2.58%) Management Information System
9 30 (2.42%) User Experience
10 28 (2.26%) Acoustics
11 24 (1.93%) Enterprise Software
12 21 (1.69%) Equities
12 21 (1.69%) Self-Motivation
13 17 (1.37%) Mobile App
14 16 (1.29%) Market Risk
15 14 (1.13%) Hybrid Cloud
16 13 (1.05%) Social Media
16 13 (1.05%) Virtual Team
17 12 (0.97%) Robotics
Operating Systems
1 164 (13.22%) Linux
2 60 (4.83%) Unix
3 25 (2.01%) Windows
4 8 (0.64%) Red Hat Enterprise Linux
5 7 (0.56%) Android
5 7 (0.56%) Apple iOS
5 7 (0.56%) VMS
6 4 (0.32%) Windows Server
7 1 (0.081%) Ubuntu
Processes & Methodologies
1 509 (41.02%) Analytics
2 500 (40.29%) Agile Software Development
3 451 (36.34%) Data Science
4 381 (30.70%) Machine Learning
5 274 (22.08%) DevOps
6 258 (20.79%) ETL
7 242 (19.50%) Data Engineering
8 224 (18.05%) Software Engineering
9 210 (16.92%) Artificial Intelligence
10 194 (15.63%) Data Modelling
11 192 (15.47%) Business Intelligence
12 187 (15.07%) Data Analytics
13 155 (12.49%) Computer Science
14 146 (11.76%) Microservices
15 145 (11.68%) Visualisation
16 128 (10.31%) Continuous Integration
17 118 (9.51%) Scrum
17 118 (9.51%) TDD
18 117 (9.43%) Fintech
19 99 (7.98%) Mentoring
Programming Languages
1 817 (65.83%) Python
2 557 (44.88%) Java
3 506 (40.77%) SQL
4 478 (38.52%) Scala
5 201 (16.20%) R
6 130 (10.48%) JavaScript
7 102 (8.22%) C#
8 95 (7.66%) C++
9 64 (5.16%) Bash Shell
10 48 (3.87%) Go
11 32 (2.58%) Java 8
12 30 (2.42%) MATLAB
13 29 (2.34%) C
13 29 (2.34%) Shell Script
14 25 (2.01%) PHP
15 24 (1.93%) Kotlin
16 19 (1.53%) Perl
17 18 (1.45%) TypeScript
18 13 (1.05%) PL/SQL
18 13 (1.05%) Ruby
Qualifications
1 253 (20.39%) Degree
2 114 (9.19%) Computer Science Degree
3 62 (5.00%) Security Cleared
4 56 (4.51%) SC Cleared
5 38 (3.06%) PhD
6 35 (2.82%) DV Cleared
7 19 (1.53%) AWS Certification
8 15 (1.21%) Master's Degree
9 12 (0.97%) Numerate Degree
10 8 (0.64%) Microsoft Certification
10 8 (0.64%) PMI Certification
10 8 (0.64%) TOGAF Certification
11 6 (0.48%) MBA
12 5 (0.40%) Postgraduate
13 4 (0.32%) CCNA
13 4 (0.32%) Cisco Certification
13 4 (0.32%) CISSP
13 4 (0.32%) Linux+ Certification
13 4 (0.32%) MCSE
13 4 (0.32%) Network+ Certification
Quality Assurance & Compliance
1 47 (3.79%) Data Quality
2 24 (1.93%) GDPR
3 23 (1.85%) QA
4 7 (0.56%) NIST
5 5 (0.40%) Sarbanes-Oxley
6 2 (0.16%) SLA
7 1 (0.081%) Actionable Recommendations
7 1 (0.081%) CDISC
7 1 (0.081%) HL7
7 1 (0.081%) MiFID
System Software
1 267 (21.51%) Docker
2 66 (5.32%) HDFS
3 42 (3.38%) Apache Flume
4 41 (3.30%) Apache ZooKeeper
5 36 (2.90%) VirtualBox
6 5 (0.40%) Active Directory
7 1 (0.081%) Alluxio
7 1 (0.081%) Ceph
7 1 (0.081%) etcd
7 1 (0.081%) Gluster
7 1 (0.081%) Hyper-V
7 1 (0.081%) VMware Infrastructure
Systems Management
1 225 (18.13%) Kubernetes
2 124 (9.99%) Ansible
3 94 (7.57%) Terraform
4 51 (4.11%) Puppet
5 38 (3.06%) Oozie
6 37 (2.98%) Docker Swarm
7 30 (2.42%) Hadoop YARN
7 30 (2.42%) Salt
8 25 (2.01%) Opscode Chef
9 18 (1.45%) Kibana
10 12 (0.97%) Mesos
11 11 (0.89%) Prometheus
12 9 (0.73%) Nagios
13 8 (0.64%) Grafana
14 4 (0.32%) Ambari
14 4 (0.32%) Chronos
14 4 (0.32%) HAProxy
14 4 (0.32%) Marathon
14 4 (0.32%) Zipkin
15 2 (0.16%) logstash
Vendors
1 166 (13.38%) Google
2 119 (9.59%) Tableau
3 98 (7.90%) Oracle
4 91 (7.33%) Databricks
5 90 (7.25%) Microsoft
6 84 (6.77%) Cloudera
7 63 (5.08%) Talend
8 61 (4.92%) Informatica
9 58 (4.67%) Hortonworks
10 56 (4.51%) IBM
11 50 (4.03%) Splunk
12 33 (2.66%) Teradata
13 17 (1.37%) CrowdStrike
14 16 (1.29%) Ab Initio
14 16 (1.29%) Netezza
14 16 (1.29%) Sybase
15 13 (1.05%) Facebook
15 13 (1.05%) StreamSets
15 13 (1.05%) Vertica
16 12 (0.97%) Dataiku