Period
to 22 September 2019

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 22 September 2019 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
22 Sep 2019
Same period 2018 Same period 2017
Rank 229 243 287
Rank change year-on-year +14 +44 +93
Permanent jobs citing Apache Spark 2,040 2,321 2,163
As % of all permanent IT jobs advertised in the UK 1.47% 1.39% 1.22%
As % of the Application Platforms category 9.90% 8.41% 6.87%
Number of salaries quoted 1,654 1,826 1,716
UK median annual salary £72,500 £70,000 £67,000
Median salary % change year-on-year +3.57% +4.48% +3.08%
10th Percentile £47,500 £45,000 £41,250
90th Percentile £110,000 £100,000 £100,000
UK excluding London median annual salary £57,500 £60,000 £57,500
% change year-on-year -4.17% +4.35% +4.55%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 20,612 27,592 31,500
As % of all permanent IT jobs advertised in the UK 14.82% 16.57% 17.83%
Number of salaries quoted 16,288 22,169 25,260
UK median annual salary £50,000 £50,000 £47,500
Median salary % change year-on-year - +5.26% +5.56%
10th Percentile £27,500 £28,000 £27,000
90th Percentile £85,000 £82,500 £76,250
UK excluding London median annual salary £42,500 £43,500 £41,000
% change year-on-year -2.30% +6.10% +2.50%

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 22 September 2019.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 22 September 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +14 1,968 £75,000 +7.14% 247
London +20 1,338 £80,000 +6.67% 177
UK excluding London +12 676 £57,500 -4.17% 87
South West +79 176 £55,000 -8.33% 12
South East -53 163 £60,000 - 21
North of England +20 128 £60,000 +9.09% 20
Midlands +22 93 £60,000 -7.69% 9
East Midlands +30 74 £57,500 -23.33% 4
North West +8 72 £65,000 +13.04% 14
East of England +9 56 £65,000 +1.96% 8
Scotland -1 56 £57,500 +4.55% 12
Yorkshire +17 45 £55,000 +8.91% 5
West Midlands +20 19 £71,250 +18.75% 5
North East +29 11 £53,638 +34.09% 1
Wales -2 3 £42,500 -26.72% 3
Channel Islands - 1 £37,500 -
Northern Ireland - 1 - - 2

For the 6 months to 22 September 2019, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,314 (64.41%) Big Data
2 1,313 (64.36%) Python
3 1,265 (62.01%) Hadoop
4 937 (45.93%) Java
5 892 (43.73%) Agile Software Development
6 847 (41.52%) SQL
7 815 (39.95%) AWS
8 813 (39.85%) Analytics
9 804 (39.41%) Scala
10 736 (36.08%) Kafka
11 734 (35.98%) Data Science
12 689 (33.77%) Apache Hive
13 642 (31.47%) Machine Learning
14 494 (24.22%) Docker
15 480 (23.53%) Azure
16 438 (21.47%) NoSQL
17 436 (21.37%) ETL
18 423 (20.74%) DevOps
19 403 (19.75%) Software Engineering
20 395 (19.36%) Linux
21 379 (18.58%) Finance
22 372 (18.24%) GCP
23 369 (18.09%) Analytical Skills
24 357 (17.50%) Data Engineering
25 346 (16.96%) R
26 333 (16.32%) Apache
27 324 (15.88%) Kubernetes
28 321 (15.74%) Git (software)
29 310 (15.20%) Degree
30 302 (14.80%) Data Analytics

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 333 (16.32%) Apache
2 236 (11.57%) Elasticsearch
3 133 (6.52%) Apache Pig
4 65 (3.19%) Apache Solr
5 64 (3.14%) Apache Airflow
6 54 (2.65%) SAS
7 43 (2.11%) OpenStack
8 26 (1.27%) Jupyter
9 25 (1.23%) Confluence
10 24 (1.18%) Hyperledger
11 22 (1.08%) Luigi
12 15 (0.74%) nginx
13 9 (0.44%) IPython
14 5 (0.25%) JBoss
14 5 (0.25%) WebSphere
15 4 (0.20%) Cloud Foundry
16 3 (0.15%) ArcGIS
16 3 (0.15%) WebLogic
17 2 (0.098%) CMS
17 2 (0.098%) Oracle Workflow
Applications
1 17 (0.83%) Microsoft Excel
2 7 (0.34%) Gephi
3 6 (0.29%) Microsoft Office
4 4 (0.20%) GNU Octave
5 2 (0.098%) Chatbot
5 2 (0.098%) Microsoft PowerPoint
5 2 (0.098%) SPSS
6 1 (0.049%) Microsoft Project
6 1 (0.049%) MS Visio
6 1 (0.049%) Weka
Business Applications
1 2 (0.098%) SAP Oil and Gas
2 1 (0.049%) Adobe Campaign
2 1 (0.049%) Magento
Cloud Services
1 815 (39.95%) AWS
2 480 (23.53%) Azure
3 372 (18.24%) GCP
4 122 (5.98%) Amazon EMR
5 87 (4.26%) Power Platform
6 85 (4.17%) Amazon S3
7 83 (4.07%) Serverless
8 76 (3.73%) Amazon Kinesis
9 68 (3.33%) PaaS
10 64 (3.14%) Cloud Computing
11 61 (2.99%) Azure Data Factory
11 61 (2.99%) SaaS
12 49 (2.40%) OpenShift
13 48 (2.35%) AWS Lambda
14 44 (2.16%) Amazon EC2
15 41 (2.01%) GitHub
16 25 (1.23%) IaaS
17 24 (1.18%) AWS CloudFormation
18 19 (0.93%) Amazon ECS
19 18 (0.88%) Cortana Intelligence Suite
Communications & Networking
1 47 (2.30%) Firewall
2 43 (2.11%) HTTPS
3 42 (2.06%) TCP/IP
4 37 (1.81%) HTTP
5 33 (1.62%) Internet
6 15 (0.74%) SSL
7 14 (0.69%) Kerberos
8 13 (0.64%) VPN
9 8 (0.39%) DNS
10 7 (0.34%) Ethernet
10 7 (0.34%) MPLS
10 7 (0.34%) Wi-Fi
11 6 (0.29%) Broadband
12 4 (0.20%) LAN
12 4 (0.20%) Software-Defined Networking
12 4 (0.20%) WAN
13 3 (0.15%) Unified Communications
14 2 (0.098%) SMS
15 1 (0.049%) Network Security
15 1 (0.049%) VXLAN
Database & Business Intelligence
1 1,314 (64.41%) Big Data
2 1,265 (62.01%) Hadoop
3 689 (33.77%) Apache Hive
4 438 (21.47%) NoSQL
5 299 (14.66%) Data Warehouse
6 278 (13.63%) Impala
7 276 (13.53%) HBase
8 256 (12.55%) Apache Cassandra
9 244 (11.96%) MongoDB
10 173 (8.48%) PostgreSQL
11 167 (8.19%) Amazon Redshift
12 154 (7.55%) Data Lake
13 146 (7.16%) Relational Database
14 129 (6.32%) SQL Server
15 115 (5.64%) Sqoop
16 108 (5.29%) MySQL
17 103 (5.05%) Data Mining
18 100 (4.90%) Flink
19 95 (4.66%) BigQuery
20 87 (4.26%) Power BI
Development Applications
1 321 (15.74%) Git (software)
2 217 (10.64%) Jenkins
3 146 (7.16%) JIRA
4 118 (5.78%) Bitbucket
5 80 (3.92%) git-flow
6 46 (2.25%) Bitbucket Server
7 35 (1.72%) Maven
8 33 (1.62%) Atlassian Bamboo
9 28 (1.37%) Selenium
10 27 (1.32%) Subversion
11 26 (1.27%) Cucumber
12 24 (1.18%) Visual Studio
13 21 (1.03%) Gradle
13 21 (1.03%) Vagrant
14 20 (0.98%) Artifactory
14 20 (0.98%) Team Foundation Server
15 19 (0.93%) Eclipse
15 19 (0.93%) IntelliJ
15 19 (0.93%) JUnit
16 17 (0.83%) Visual Studio Team System
General
1 379 (18.58%) Finance
2 143 (7.01%) Retail
3 125 (6.13%) Banking
4 93 (4.56%) Law
5 90 (4.41%) Marketing
6 71 (3.48%) Telecoms
7 46 (2.25%) Public Sector
8 43 (2.11%) Games
9 41 (2.01%) Manufacturing
10 37 (1.81%) Front Office
11 33 (1.62%) Advertising
12 29 (1.42%) Investment Banking
12 29 (1.42%) Publishing
13 28 (1.37%) Financial Institution
14 26 (1.27%) Automotive
15 18 (0.88%) Electronics
16 14 (0.69%) Legal
17 9 (0.44%) Billing
18 8 (0.39%) Aerospace
18 8 (0.39%) Retail Banking
Job Titles
1 623 (30.54%) Data Engineer
2 479 (23.48%) Developer
3 251 (12.30%) Java Developer
4 243 (11.91%) Data Scientist
5 208 (10.20%) Big Data Engineer
6 163 (7.99%) Architect
7 161 (7.89%) Senior Data Engineer
8 125 (6.13%) Software Engineer
9 122 (5.98%) Big Data Developer
10 117 (5.74%) Data Architect
11 115 (5.64%) Python Engineer
12 113 (5.54%) Senior Developer
13 107 (5.25%) Senior Data Scientist
14 100 (4.90%) Consultant
15 97 (4.75%) Scala Developer
16 73 (3.58%) Senior Java Developer
17 70 (3.43%) Hadoop Engineer
18 68 (3.33%) Python Developer
19 67 (3.28%) Lead Developer
20 66 (3.24%) Data Consultant
Libraries, Frameworks & Software Standards
1 736 (36.08%) Kafka
2 164 (8.04%) Spring
3 161 (7.89%) REST
4 155 (7.60%) Web Services
5 127 (6.23%) Pandas
6 122 (5.98%) scikit-learn
7 119 (5.83%) TensorFlow
8 110 (5.39%) React
9 104 (5.10%) Node.js
10 97 (4.75%) AngularJS
11 96 (4.71%) Elastic Stack
12 95 (4.66%) Natural Language Processing
13 93 (4.56%) Akka
14 82 (4.02%) RESTful
15 78 (3.82%) NumPy
16 76 (3.73%) Apache NiFi
16 76 (3.73%) Spark Streaming
17 73 (3.58%) .NET
18 67 (3.28%) JSON
19 64 (3.14%) GraphQL
Miscellaneous
1 369 (18.09%) Analytical Skills
2 293 (14.36%) Algorithms
3 104 (5.10%) Data Structures
4 76 (3.73%) Greenfield Project
5 54 (2.65%) Analytical Mindset
6 49 (2.40%) Public Cloud
7 43 (2.11%) Cyber Kill Chain
8 39 (1.91%) Management Information System
9 36 (1.76%) Social Media
10 34 (1.67%) Private Cloud
11 32 (1.57%) Virtual Team
12 31 (1.52%) Linux Command Line
13 29 (1.42%) Cybercrime
13 29 (1.42%) Self-Motivation
14 26 (1.27%) User Experience
15 25 (1.23%) Derivative
16 22 (1.08%) Blog
16 22 (1.08%) Cloud Native
17 20 (0.98%) Mobile App
18 18 (0.88%) Cyberattack
Operating Systems
1 395 (19.36%) Linux
2 167 (8.19%) Unix
3 52 (2.55%) Windows
4 41 (2.01%) Android
4 41 (2.01%) Apple iOS
5 13 (0.64%) CentOS
6 8 (0.39%) Ubuntu
7 6 (0.29%) Red Hat Enterprise Linux
7 6 (0.29%) Windows Server
8 2 (0.098%) SUSE Linux Enterprise Server
9 1 (0.049%) Debian
Processes & Methodologies
1 892 (43.73%) Agile Software Development
2 813 (39.85%) Analytics
3 734 (35.98%) Data Science
4 642 (31.47%) Machine Learning
5 436 (21.37%) ETL
6 423 (20.74%) DevOps
7 403 (19.75%) Software Engineering
8 357 (17.50%) Data Engineering
9 302 (14.80%) Data Analytics
10 301 (14.75%) Business Intelligence
11 291 (14.26%) Data Modelling
12 288 (14.12%) Visualisation
13 276 (13.53%) Scrum
14 268 (13.14%) Artificial Intelligence
15 245 (12.01%) Microservices
16 231 (11.32%) MapReduce
17 227 (11.13%) Mathematics
18 220 (10.78%) Computer Science
19 189 (9.26%) Open Source
20 185 (9.07%) Problem-Solving
Programming Languages
1 1,313 (64.36%) Python
2 937 (45.93%) Java
3 847 (41.52%) SQL
4 804 (39.41%) Scala
5 346 (16.96%) R
6 174 (8.53%) JavaScript
7 127 (6.23%) Bash Shell
8 122 (5.98%) C
9 121 (5.93%) C#
10 81 (3.97%) C++
11 60 (2.94%) Perl
12 57 (2.79%) Go
12 57 (2.79%) Java 8
13 49 (2.40%) PL/SQL
13 49 (2.40%) Shell Script
14 33 (1.62%) TypeScript
15 28 (1.37%) Ruby
16 21 (1.03%) T-SQL
17 20 (0.98%) PHP
18 19 (0.93%) PowerShell
Qualifications
1 310 (15.20%) Degree
2 106 (5.20%) Security Cleared
3 102 (5.00%) PhD
4 90 (4.41%) Computer Science Degree
5 49 (2.40%) Master's Degree
6 47 (2.30%) SC Cleared
7 45 (2.21%) DV Cleared
8 44 (2.16%) Numerate Degree
9 41 (2.01%) Microsoft Certification
10 40 (1.96%) CCNA
10 40 (1.96%) Cisco Certification
11 39 (1.91%) CISSP
11 39 (1.91%) CompTIA Security+
11 39 (1.91%) GIAC
11 39 (1.91%) Linux+ Certification
11 39 (1.91%) MCSE
11 39 (1.91%) Network+ Certification
11 39 (1.91%) SANS
11 39 (1.91%) SSCP
12 15 (0.74%) PMP
Quality Assurance & Compliance
1 94 (4.61%) Data Quality
2 70 (3.43%) QA
3 16 (0.78%) GDPR
4 6 (0.29%) ISO/IEC 27001
5 3 (0.15%) Actionable Recommendations
6 2 (0.098%) ISO 15022
6 2 (0.098%) ISO 20022
6 2 (0.098%) PMO
7 1 (0.049%) EMIR
7 1 (0.049%) IFRS
7 1 (0.049%) IFRS 9
7 1 (0.049%) MiFID
System Software
1 494 (24.22%) Docker
2 165 (8.09%) HDFS
3 113 (5.54%) Apache Flume
4 68 (3.33%) Apache ZooKeeper
5 20 (0.98%) VirtualBox
6 10 (0.49%) Cronacle
7 4 (0.20%) Active Directory
8 3 (0.15%) VMware Infrastructure
9 1 (0.049%) Ceph
9 1 (0.049%) Hyper-V
9 1 (0.049%) Virtual Servers
Systems Management
1 324 (15.88%) Kubernetes
2 167 (8.19%) Ansible
3 164 (8.04%) Oozie
4 146 (7.16%) Hadoop YARN
4 146 (7.16%) Terraform
5 67 (3.28%) Puppet
6 52 (2.55%) Opscode Chef
7 33 (1.62%) Ambari
8 29 (1.42%) Docker Swarm
9 22 (1.08%) Kibana
9 22 (1.08%) Mesos
10 16 (0.78%) Consul
11 14 (0.69%) Control-M
11 14 (0.69%) Marathon
12 9 (0.44%) logstash
12 9 (0.44%) Nagios
13 8 (0.39%) Salt
14 7 (0.34%) Rancher
14 7 (0.34%) Single Sign-On
15 6 (0.29%) Prometheus
Vendors
1 283 (13.87%) Google
2 180 (8.82%) Cloudera
3 160 (7.84%) Tableau
4 150 (7.35%) Oracle
5 138 (6.76%) Informatica
6 134 (6.57%) Hortonworks
7 111 (5.44%) Databricks
8 109 (5.34%) Microsoft
9 93 (4.56%) Splunk
10 64 (3.14%) Talend
11 62 (3.04%) Teradata
12 43 (2.11%) IBM
13 39 (1.91%) SAP
14 36 (1.76%) Ab Initio
15 31 (1.52%) Netezza
16 30 (1.47%) MapR
17 22 (1.08%) WorldPay
18 20 (0.98%) Dataiku
19 17 (0.83%) VMware
20 15 (0.74%) Vertica