Period
to 8 December 2019

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 8 December 2019 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
8 Dec 2019
Same period 2018 Same period 2017
Rank 229 242 267
Rank change year-on-year +13 +25 +80
Permanent jobs citing Apache Spark 1,900 2,231 2,375
As % of all permanent IT jobs advertised in the UK 1.43% 1.40% 1.32%
As % of the Application Platforms category 9.72% 8.71% 7.63%
Number of salaries quoted 1,578 1,740 1,940
UK median annual salary £72,500 £72,500 £67,000
Median salary % change year-on-year - +8.21% -0.74%
10th Percentile £48,175 £45,000 £42,225
90th Percentile £105,000 £102,500 £102,500
UK excluding London median annual salary £60,000 £65,000 £52,500
% change year-on-year -7.69% +23.81% -4.55%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 19,545 25,600 31,111
As % of all permanent IT jobs advertised in the UK 14.71% 16.11% 17.30%
Number of salaries quoted 15,760 20,563 25,249
UK median annual salary £50,000 £50,000 £47,500
Median salary % change year-on-year - +5.26% +5.56%
10th Percentile £27,500 £28,000 £27,000
90th Percentile £85,000 £83,750 £77,500
UK excluding London median annual salary £44,000 £43,500 £40,000
% change year-on-year +1.15% +8.75% -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 8 December 2019.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 8 December 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +21 1,839 £72,500 - 294
London +18 1,244 £77,500 - 206
UK excluding London +4 636 £60,000 -7.69% 101
North of England +36 147 £60,000 -14.29% 24
South East -54 145 £60,000 -7.69% 26
South West +20 123 £55,000 -8.33% 4
Midlands +36 106 £70,000 -6.67% 12
East Midlands +35 87 £72,500 -3.33% 7
Yorkshire +53 69 £57,275 +13.42% 7
North West -1 66 £65,000 -7.14% 17
East of England +51 60 £72,500 +9.85% 21
Scotland -4 54 £56,250 +2.27% 14
West Midlands +17 19 £60,000 +41.18% 5
North East +19 12 £54,671 -45.33%
Wales -11 1 £42,500 -26.72%

For the 6 months to 8 December 2019, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,174 (61.79%) Python
2 1,146 (60.32%) Big Data
3 1,140 (60.00%) Hadoop
4 822 (43.26%) Java
5 798 (42.00%) Agile Software Development
6 792 (41.68%) AWS
7 739 (38.89%) SQL
8 716 (37.68%) Analytics
9 703 (37.00%) Scala
10 685 (36.05%) Data Science
11 642 (33.79%) Kafka
12 589 (31.00%) Machine Learning
13 565 (29.74%) Apache Hive
14 483 (25.42%) Azure
15 461 (24.26%) Docker
16 398 (20.95%) ETL
16 398 (20.95%) NoSQL
17 376 (19.79%) DevOps
18 369 (19.42%) Finance
19 366 (19.26%) Software Engineering
20 360 (18.95%) GCP
21 357 (18.79%) Kubernetes
22 353 (18.58%) Data Engineering
23 352 (18.53%) Linux
24 337 (17.74%) Git (software)
25 327 (17.21%) Analytical Skills
26 309 (16.26%) Apache
27 287 (15.11%) Business Intelligence
28 283 (14.89%) Data Modelling
29 282 (14.84%) Visualisation

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 309 (16.26%) Apache
2 190 (10.00%) Elasticsearch
3 76 (4.00%) Apache Airflow
4 45 (2.37%) Apache Solr
5 41 (2.16%) Apache Pig
6 39 (2.05%) SAS
7 35 (1.84%) Luigi
8 26 (1.37%) Jupyter
9 24 (1.26%) OpenStack
10 22 (1.16%) Confluence
11 20 (1.05%) Hyperledger
12 15 (0.79%) nginx
13 5 (0.26%) SharePoint
14 4 (0.21%) DataSynapse
14 4 (0.21%) IPython
15 3 (0.16%) ArcGIS
15 3 (0.16%) Cloud Foundry
15 3 (0.16%) JBoss
16 2 (0.11%) CMS
16 2 (0.11%) Tomcat
Applications
1 7 (0.37%) Gephi
1 7 (0.37%) Microsoft Excel
2 3 (0.16%) Chatbot
2 3 (0.16%) GNU Octave
3 2 (0.11%) Microsoft Office
3 2 (0.11%) Microsoft PowerPoint
3 2 (0.11%) SPSS
4 1 (0.053%) Microsoft Project
4 1 (0.053%) MS Visio
4 1 (0.053%) Weka
Business Applications
1 4 (0.21%) Sparx Enterprise Architect
2 2 (0.11%) Eikon
3 1 (0.053%) Adobe Campaign
3 1 (0.053%) Magento
3 1 (0.053%) NetSuite
3 1 (0.053%) SAP Oil and Gas
Cloud Services
1 792 (41.68%) AWS
2 483 (25.42%) Azure
3 360 (18.95%) GCP
4 118 (6.21%) Amazon EMR
5 115 (6.05%) Amazon S3
6 85 (4.47%) Serverless
7 81 (4.26%) Power Platform
8 70 (3.68%) Amazon Kinesis
9 69 (3.63%) PaaS
10 64 (3.37%) AWS Lambda
11 50 (2.63%) Cloud Computing
12 48 (2.53%) GitHub
13 46 (2.42%) SaaS
14 42 (2.21%) OpenShift
15 31 (1.63%) Amazon EC2
15 31 (1.63%) Azure Data Factory
16 29 (1.53%) Amazon SageMaker
17 28 (1.47%) IaaS
18 24 (1.26%) Azure Machine Learning
19 22 (1.16%) IBM Cloud
Communications & Networking
1 43 (2.26%) HTTP
2 34 (1.79%) Firewall
3 29 (1.53%) HTTPS
4 22 (1.16%) TCP/IP
5 20 (1.05%) VPN
6 18 (0.95%) Internet
7 17 (0.89%) Kerberos
7 17 (0.89%) SSL
8 8 (0.42%) DNS
8 8 (0.42%) Ethernet
8 8 (0.42%) MPLS
9 4 (0.21%) FTP
10 3 (0.16%) 3G
10 3 (0.16%) 4G
10 3 (0.16%) 5G
10 3 (0.16%) Wi-Fi
11 2 (0.11%) SMS
12 1 (0.053%) Intrusion Detection
12 1 (0.053%) Network Security
12 1 (0.053%) Unified Communications
Database & Business Intelligence
1 1,146 (60.32%) Big Data
2 1,140 (60.00%) Hadoop
3 565 (29.74%) Apache Hive
4 398 (20.95%) NoSQL
5 268 (14.11%) Data Warehouse
6 217 (11.42%) HBase
7 214 (11.26%) Impala
8 211 (11.11%) Apache Cassandra
9 208 (10.95%) PostgreSQL
10 195 (10.26%) MongoDB
11 165 (8.68%) Relational Database
12 159 (8.37%) Amazon Redshift
13 136 (7.16%) Data Lake
14 134 (7.05%) SQL Server
15 118 (6.21%) BigQuery
16 109 (5.74%) Flink
17 91 (4.79%) Neo4j
18 89 (4.68%) Data Mining
19 81 (4.26%) Power BI
20 79 (4.16%) Sqoop
Development Applications
1 337 (17.74%) Git (software)
2 212 (11.16%) Jenkins
3 150 (7.89%) JIRA
4 80 (4.21%) Bitbucket
5 58 (3.05%) git-flow
6 51 (2.68%) Atlassian Bamboo
7 35 (1.84%) Bitbucket Server
8 34 (1.79%) Vagrant
9 28 (1.47%) Maven
10 27 (1.42%) JUnit
11 23 (1.21%) Gradle
12 22 (1.16%) Eclipse
13 21 (1.11%) IntelliJ
14 19 (1.00%) Cucumber
14 19 (1.00%) Selenium
15 18 (0.95%) Subversion
16 15 (0.79%) Visual Studio Team System
17 14 (0.74%) GitLab
18 12 (0.63%) Team Foundation Server
19 10 (0.53%) TestNG
General
1 369 (19.42%) Finance
2 126 (6.63%) Retail
3 109 (5.74%) Banking
4 72 (3.79%) Marketing
5 60 (3.16%) Telecoms
6 56 (2.95%) Public Sector
7 41 (2.16%) Manufacturing
8 38 (2.00%) Front Office
9 35 (1.84%) Games
10 31 (1.63%) Automotive
11 27 (1.42%) Advertising
11 27 (1.42%) Legal
12 23 (1.21%) Investment Banking
13 22 (1.16%) Financial Institution
14 14 (0.74%) Publishing
15 13 (0.68%) Electronics
15 13 (0.68%) Law
16 9 (0.47%) Retail Banking
17 6 (0.32%) Billing
17 6 (0.32%) Online Betting
Job Titles
1 571 (30.05%) Data Engineer
2 446 (23.47%) Developer
3 247 (13.00%) Java Developer
4 213 (11.21%) Data Scientist
5 189 (9.95%) Architect
6 136 (7.16%) Data Architect
7 126 (6.63%) Big Data Engineer
8 120 (6.32%) Senior Data Engineer
9 111 (5.84%) Senior Developer
10 107 (5.63%) Senior Data Scientist
11 106 (5.58%) Software Engineer
12 92 (4.84%) Python Engineer
13 88 (4.63%) Scala Developer
14 84 (4.42%) Consultant
15 70 (3.68%) Senior Java Developer
16 66 (3.47%) Big Data Developer
17 59 (3.11%) Data Consultant
18 56 (2.95%) Python Developer
19 53 (2.79%) Java Engineer
20 48 (2.53%) Analyst
Libraries, Frameworks & Software Standards
1 642 (33.79%) Kafka
2 171 (9.00%) REST
3 134 (7.05%) TensorFlow
4 131 (6.89%) Spring
5 128 (6.74%) Web Services
6 118 (6.21%) scikit-learn
7 116 (6.11%) Natural Language Processing
8 111 (5.84%) Node.js
8 111 (5.84%) Pandas
9 109 (5.74%) React
10 98 (5.16%) AngularJS
11 81 (4.26%) RESTful
12 76 (4.00%) Akka
13 72 (3.79%) Elastic Stack
14 70 (3.68%) HTML
15 64 (3.37%) Spark Streaming
16 63 (3.32%) NumPy
17 58 (3.05%) Keras
18 57 (3.00%) JSON
19 55 (2.89%) GraphQL
Miscellaneous
1 327 (17.21%) Analytical Skills
2 262 (13.79%) Algorithms
3 94 (4.95%) Data Structures
4 76 (4.00%) Greenfield Project
5 67 (3.53%) Public Cloud
6 39 (2.05%) Self-Motivation
7 37 (1.95%) Cyber Kill Chain
7 37 (1.95%) Management Information System
8 36 (1.89%) Analytical Mindset
9 32 (1.68%) User Experience
10 31 (1.63%) Linux Command Line
11 29 (1.53%) Virtual Team
12 27 (1.42%) Blog
12 27 (1.42%) Private Cloud
13 25 (1.32%) Cloud Native
14 24 (1.26%) Social Media
15 22 (1.16%) Data Centre
15 22 (1.16%) Derivative
16 20 (1.05%) Cybercrime
16 20 (1.05%) Video on Demand
Operating Systems
1 352 (18.53%) Linux
2 121 (6.37%) Unix
3 27 (1.42%) Windows
4 22 (1.16%) Android
4 22 (1.16%) Apple iOS
5 5 (0.26%) Red Hat Enterprise Linux
6 2 (0.11%) SUSE Linux Enterprise Server
7 1 (0.053%) CoreOS
Processes & Methodologies
1 798 (42.00%) Agile Software Development
2 716 (37.68%) Analytics
3 685 (36.05%) Data Science
4 589 (31.00%) Machine Learning
5 398 (20.95%) ETL
6 376 (19.79%) DevOps
7 366 (19.26%) Software Engineering
8 353 (18.58%) Data Engineering
9 287 (15.11%) Business Intelligence
10 283 (14.89%) Data Modelling
11 282 (14.84%) Visualisation
12 268 (14.11%) Artificial Intelligence
13 260 (13.68%) Data Analytics
14 255 (13.42%) Computer Science
15 252 (13.26%) Microservices
16 235 (12.37%) Scrum
17 187 (9.84%) Mathematics
18 181 (9.53%) Statistics
19 172 (9.05%) Mentoring
20 165 (8.68%) Open Source
Programming Languages
1 1,174 (61.79%) Python
2 822 (43.26%) Java
3 739 (38.89%) SQL
4 703 (37.00%) Scala
5 276 (14.53%) R
6 206 (10.84%) JavaScript
7 115 (6.05%) Bash Shell
8 113 (5.95%) C
8 113 (5.95%) C++
9 112 (5.89%) C#
10 87 (4.58%) Go
11 55 (2.89%) Shell Script
12 51 (2.68%) Java 8
13 46 (2.42%) Perl
14 44 (2.32%) TypeScript
15 42 (2.21%) PL/SQL
16 34 (1.79%) PHP
17 32 (1.68%) Kotlin
18 21 (1.11%) Ruby
19 19 (1.00%) MATLAB
Qualifications
1 278 (14.63%) Degree
2 96 (5.05%) PhD
3 95 (5.00%) Computer Science Degree
4 76 (4.00%) Security Cleared
5 45 (2.37%) Master's Degree
6 40 (2.11%) DV Cleared
7 38 (2.00%) SC Cleared
8 32 (1.68%) Numerate Degree
9 24 (1.26%) Microsoft Certification
10 23 (1.21%) CCNA
10 23 (1.21%) Cisco Certification
11 22 (1.16%) CISSP
11 22 (1.16%) CompTIA Security+
11 22 (1.16%) GIAC
11 22 (1.16%) Linux+ Certification
11 22 (1.16%) MCSE
11 22 (1.16%) Network+ Certification
11 22 (1.16%) SANS
11 22 (1.16%) SSCP
12 7 (0.37%) PMI Certification
Quality Assurance & Compliance
1 104 (5.47%) Data Quality
2 80 (4.21%) QA
3 25 (1.32%) GDPR
4 12 (0.63%) ISO/IEC 27001
5 3 (0.16%) Actionable Recommendations
5 3 (0.16%) CDISC
5 3 (0.16%) EMIR
5 3 (0.16%) HL7
5 3 (0.16%) ISO 26262
5 3 (0.16%) MiFID
6 2 (0.11%) PMO
7 1 (0.053%) IFRS
7 1 (0.053%) IFRS 9
System Software
1 461 (24.26%) Docker
2 154 (8.11%) HDFS
3 82 (4.32%) Apache Flume
4 34 (1.79%) VirtualBox
5 30 (1.58%) Apache ZooKeeper
6 8 (0.42%) Cronacle
7 3 (0.16%) Varnish
8 1 (0.053%) Active Directory
8 1 (0.053%) Hyper-V
8 1 (0.053%) Virtual Servers
8 1 (0.053%) VMware Infrastructure
Systems Management
1 357 (18.79%) Kubernetes
2 165 (8.68%) Ansible
3 155 (8.16%) Terraform
4 138 (7.26%) Oozie
5 95 (5.00%) Hadoop YARN
6 53 (2.79%) Puppet
7 41 (2.16%) Docker Swarm
8 23 (1.21%) Ambari
9 20 (1.05%) Consul
10 17 (0.89%) Kibana
10 17 (0.89%) Opscode Chef
11 16 (0.84%) Control-M
12 13 (0.68%) Salt
13 12 (0.63%) Single Sign-On
14 11 (0.58%) Mesos
15 9 (0.47%) Prometheus
16 8 (0.42%) logstash
16 8 (0.42%) Rancher
17 6 (0.32%) Grafana
17 6 (0.32%) Marathon
Vendors
1 281 (14.79%) Google
2 144 (7.58%) Cloudera
3 138 (7.26%) Tableau
4 125 (6.58%) Hortonworks
5 114 (6.00%) Oracle
6 105 (5.53%) Informatica
7 92 (4.84%) Microsoft
8 82 (4.32%) Databricks
9 77 (4.05%) Splunk
10 69 (3.63%) Talend
11 54 (2.84%) Teradata
12 51 (2.68%) IBM
13 28 (1.47%) WorldPay
14 24 (1.26%) Ab Initio
15 23 (1.21%) SAP
16 21 (1.11%) Dataiku
16 21 (1.11%) MapR
17 16 (0.84%) VMware
18 15 (0.79%) FIS
18 15 (0.79%) Netezza