Period
to 20 July 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 20 July 2018 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
20 Jul 2018
Same period 2017 Same period 2016
Rank 234 291 382
Rank change year-on-year +57 +91 +454
Permanent jobs citing Apache Spark 2,579 2,106 1,676
As % of all permanent IT jobs advertised in the UK 1.47% 1.19% 0.83%
As % of the Application Platforms category 8.87% 6.58% 4.40%
Number of salaries quoted 1,977 1,675 1,379
UK median annual salary £70,000 £67,000 £65,000
Median salary % change year-on-year +4.48% +3.08% -
10th Percentile £43,750 £41,250 £40,000
90th Percentile £100,000 £98,450 £103,750
UK excluding London median annual salary £57,500 £57,500 £57,500

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 29,060 31,995 38,121
As % of all permanent IT jobs advertised in the UK 16.60% 18.04% 18.78%
Number of salaries quoted 23,284 26,034 32,297
UK median annual salary £50,000 £47,500 £45,000
Median salary % change year-on-year +5.26% +5.56% -
10th Percentile £28,000 £27,000 £26,325
90th Percentile £80,000 £76,250 £73,750
UK excluding London median annual salary £42,500 £40,000 £40,000
% change year-on-year +6.25% - -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 20 July 2018.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 20 July 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +55 2,464 £70,000 +3.70% 276
London +63 1,723 £75,000 +7.14% 188
UK excluding London +8 816 £57,500 - 115
South East +32 259 £60,000 -7.69% 31
North of England -10 170 £55,000 - 27
East of England +13 136 £66,000 +32.00% 11
North West -6 107 £55,000 -8.33% 18
South West +19 89 £55,000 +4.76% 11
Midlands +9 80 £60,000 +23.71% 8
Scotland -3 72 £55,000 -29.03% 23
Yorkshire +32 58 £55,000 - 9
East Midlands +6 42 £60,000 +41.18% 6
West Midlands +32 38 £60,000 - 2
Wales -2 11 £48,000 +37.14% 1
North East -32 5 £40,000 +14.29%

For the 6 months to 20 July 2018, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,836 (71.19%) Hadoop
2 1,717 (66.58%) Big Data
3 1,531 (59.36%) Python
4 1,272 (49.32%) Java
5 1,049 (40.67%) Analytics
6 1,044 (40.48%) Scala
7 1,005 (38.97%) SQL
8 992 (38.46%) Agile Software Development
9 909 (35.25%) Kafka
10 859 (33.31%) Machine Learning
11 858 (33.27%) Data Science
12 794 (30.79%) Amazon AWS
13 718 (27.84%) Apache Hive
14 653 (25.32%) NoSQL
15 529 (20.51%) R
16 492 (19.08%) Finance
16 492 (19.08%) ETL
17 465 (18.03%) Apache Cassandra
18 451 (17.49%) Degree
19 438 (16.98%) Linux
20 409 (15.86%) Software Engineering
21 408 (15.82%) DevOps
22 407 (15.78%) Algorithms
23 393 (15.24%) Analytical Skills
24 384 (14.89%) Microsoft Azure
25 382 (14.81%) Computer Science
25 382 (14.81%) TDD
26 375 (14.54%) Problem-Solving
27 370 (14.35%) Artificial Intelligence
28 364 (14.11%) Apache

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 364 (14.11%) Apache
2 271 (10.51%) Elasticsearch
3 186 (7.21%) Apache Pig
4 118 (4.58%) Apache Solr
5 114 (4.42%) SAS
6 35 (1.36%) Confluence
7 30 (1.16%) Tomcat
8 26 (1.01%) CMS
9 18 (0.70%) Jupyter
10 16 (0.62%) OpenStack
11 15 (0.58%) Apache Airflow
11 15 (0.58%) nginx
12 14 (0.54%) JBoss
13 13 (0.50%) Oracle SOA Suite
14 11 (0.43%) WildFly
15 8 (0.31%) GemFire
15 8 (0.31%) Jetty
16 7 (0.27%) NServiceBus
17 6 (0.23%) Oracle Workflow
18 5 (0.19%) WebLogic
Applications
1 39 (1.51%) Microsoft Excel
2 21 (0.81%) Chatbot
3 16 (0.62%) GNU Octave
4 12 (0.47%) SPSS
5 7 (0.27%) MS Visio
6 6 (0.23%) Microsoft PowerPoint
6 6 (0.23%) Weka
7 3 (0.12%) Mathematica
7 3 (0.12%) Microsoft Office
8 2 (0.078%) Microsoft Project
9 1 (0.039%) CPLEX
9 1 (0.039%) Mingle
9 1 (0.039%) SPSS Modeler
Business Applications
1 3 (0.12%) Distributed Ledger
2 1 (0.039%) Actimize
2 1 (0.039%) Adobe Campaign
2 1 (0.039%) Eloqua
2 1 (0.039%) NetSuite
2 1 (0.039%) Oracle 11i
2 1 (0.039%) Oracle AML
2 1 (0.039%) Oracle Assets
2 1 (0.039%) Oracle EBS
2 1 (0.039%) Oracle EBS R12
2 1 (0.039%) Oracle General Ledger
2 1 (0.039%) Oracle iProcurement
2 1 (0.039%) Oracle Payables
2 1 (0.039%) Oracle Receivables
2 1 (0.039%) WebTrends
Cloud Services
1 794 (30.79%) Amazon AWS
2 384 (14.89%) Microsoft Azure
3 167 (6.48%) Amazon EMR
4 166 (6.44%) Google Cloud Platform
5 128 (4.96%) Amazon S3
6 110 (4.27%) Amazon Kinesis
7 102 (3.96%) Serverless
8 80 (3.10%) AWS Lambda
8 80 (3.10%) GitHub
9 58 (2.25%) Amazon EC2
10 54 (2.09%) Cloud Computing
11 46 (1.78%) SaaS
12 38 (1.47%) IBM Cloud
13 34 (1.32%) PaaS
14 26 (1.01%) Azure Machine Learning
15 24 (0.93%) IaaS
16 19 (0.74%) Google Analytics
17 18 (0.70%) AWS CloudFormation
18 16 (0.62%) Slack
19 15 (0.58%) Amazon ELB
Communications & Networking
1 50 (1.94%) HTTP
2 30 (1.16%) Kerberos
3 25 (0.97%) Internet
4 21 (0.81%) Firewall
5 20 (0.78%) DNS
5 20 (0.78%) Sourcefire
6 19 (0.74%) TCP/IP
7 12 (0.47%) Network Security
7 12 (0.47%) SMS
7 12 (0.47%) VPN
8 9 (0.35%) SMTP
9 5 (0.19%) IPsec
9 5 (0.19%) MQTT
10 4 (0.16%) DAB
10 4 (0.16%) SSL
11 3 (0.12%) DHCP
11 3 (0.12%) DSL
11 3 (0.12%) FTP
12 2 (0.078%) UMTS
12 2 (0.078%) Wireless
Database & Business Intelligence
1 1,836 (71.19%) Hadoop
2 1,717 (66.58%) Big Data
3 718 (27.84%) Apache Hive
4 653 (25.32%) NoSQL
5 465 (18.03%) Apache Cassandra
6 355 (13.77%) HBase
7 354 (13.73%) MongoDB
8 263 (10.20%) Impala
9 253 (9.81%) Data Warehouse
10 251 (9.73%) Relational Database
11 242 (9.38%) Amazon Redshift
12 188 (7.29%) Data Lake
13 165 (6.40%) SQL Server
14 141 (5.47%) MySQL
15 132 (5.12%) PostgreSQL
16 119 (4.61%) RDBMS
17 115 (4.46%) Power BI
18 111 (4.30%) Sqoop
19 100 (3.88%) Data Mining
20 92 (3.57%) Redis
Development Applications
1 211 (8.18%) Jenkins
2 177 (6.86%) Git (software)
3 93 (3.61%) JIRA
4 81 (3.14%) Maven
5 70 (2.71%) JUnit
6 56 (2.17%) Atlassian Bamboo
7 55 (2.13%) Gradle
8 50 (1.94%) Cucumber
9 48 (1.86%) Selenium
10 32 (1.24%) Subversion
10 32 (1.24%) Vagrant
11 31 (1.20%) TeamCity
12 26 (1.01%) RStudio
12 26 (1.01%) Sonatype Nexus
13 23 (0.89%) Mockito
14 19 (0.74%) JMeter
15 17 (0.66%) Artifactory
15 17 (0.66%) Bitbucket
15 17 (0.66%) gulp
16 16 (0.62%) GitLab
General
1 492 (19.08%) Finance
2 252 (9.77%) Banking
3 186 (7.21%) Retail
4 152 (5.89%) Investment Banking
5 139 (5.39%) Marketing
6 102 (3.96%) Telecoms
7 96 (3.72%) Financial Institution
8 95 (3.68%) Front Office
8 95 (3.68%) Games
9 53 (2.06%) Advertising
10 52 (2.02%) Electronics
11 45 (1.74%) Manufacturing
12 39 (1.51%) Pharmaceutical
13 31 (1.20%) Law
14 22 (0.85%) Publishing
15 10 (0.39%) Military
16 9 (0.35%) Automotive
17 6 (0.23%) Legal
18 5 (0.19%) Music Industry
19 4 (0.16%) Retail Banking
Job Titles
1 606 (23.50%) Developer
2 592 (22.95%) Data Engineer
3 433 (16.79%) Data Scientist
4 320 (12.41%) Big Data Engineer
5 294 (11.40%) Architect
6 255 (9.89%) Java Developer
7 190 (7.37%) Data Architect
8 164 (6.36%) Senior Data Scientist
9 153 (5.93%) Big Data Developer
10 127 (4.92%) Scala Developer
11 126 (4.89%) Big Data Architect
12 122 (4.73%) Senior Developer
13 106 (4.11%) Software Engineer
14 105 (4.07%) Hadoop Engineer
15 87 (3.37%) Java Engineer
16 86 (3.33%) Senior Data Engineer
17 76 (2.95%) AWS Engineer
18 74 (2.87%) DevOps Engineer
19 70 (2.71%) Python Engineer
20 69 (2.68%) Machine Learning Scientist
Libraries, Frameworks & Software Standards
1 909 (35.25%) Kafka
2 203 (7.87%) Web Services
3 187 (7.25%) Natural Language Processing
4 177 (6.86%) Spring
5 171 (6.63%) .NET
6 170 (6.59%) AngularJS
7 136 (5.27%) Node.js
8 135 (5.23%) Pandas
8 135 (5.23%) REST
9 131 (5.08%) TensorFlow
10 123 (4.77%) RESTful
11 122 (4.73%) HTML
12 113 (4.38%) Spark Streaming
13 106 (4.11%) scikit-learn
14 104 (4.03%) Akka
15 100 (3.88%) React
16 93 (3.61%) NumPy
17 88 (3.41%) RabbitMQ
18 85 (3.30%) HTML5
19 84 (3.26%) Hibernate
Miscellaneous
1 407 (15.78%) Algorithms
2 393 (15.24%) Analytical Skills
3 382 (14.81%) Computer Science
4 161 (6.24%) Fintech
5 156 (6.05%) Data Structures
6 125 (4.85%) Greenfield Project
7 85 (3.30%) Cyberthreat
8 76 (2.95%) Self-Motivation
9 73 (2.83%) User Experience
10 57 (2.21%) Management Information System
11 44 (1.71%) Virtual Team
12 40 (1.55%) Linux Command Line
13 38 (1.47%) Derivative
14 33 (1.28%) Equities
15 30 (1.16%) Fixed Income
15 30 (1.16%) Hedge funds
16 28 (1.09%) Mobile App
17 26 (1.01%) Foreign Exchange (FX)
18 25 (0.97%) Public Cloud
19 24 (0.93%) Clustering
Operating Systems
1 438 (16.98%) Linux
2 177 (6.86%) Unix
3 70 (2.71%) Windows
4 23 (0.89%) Ubuntu
5 17 (0.66%) Android
6 16 (0.62%) Apple iOS
6 16 (0.62%) CentOS
6 16 (0.62%) Red Hat Enterprise Linux
7 6 (0.23%) CoreOS
8 5 (0.19%) Amazon Linux
8 5 (0.19%) VMS
9 4 (0.16%) Windows Server
10 3 (0.12%) Debian
10 3 (0.12%) Windows 10
11 2 (0.078%) Mac OS
12 1 (0.039%) AIX
12 1 (0.039%) Mac OS X
12 1 (0.039%) Mesosphere
12 1 (0.039%) Solaris
12 1 (0.039%) Windows Server 2008
Processes & Methodologies
1 1,049 (40.67%) Analytics
2 992 (38.46%) Agile Software Development
3 859 (33.31%) Machine Learning
4 858 (33.27%) Data Science
5 492 (19.08%) ETL
6 409 (15.86%) Software Engineering
7 408 (15.82%) DevOps
8 382 (14.81%) TDD
9 375 (14.54%) Problem-Solving
10 370 (14.35%) Artificial Intelligence
11 338 (13.11%) Mathematics
12 320 (12.41%) Visualisation
13 312 (12.10%) Data Analytics
14 304 (11.79%) Scrum
15 296 (11.48%) Business Intelligence
16 291 (11.28%) Open Source
17 279 (10.82%) Continuous Integration
18 277 (10.74%) Data Modelling
19 246 (9.54%) Data Engineering
20 204 (7.91%) Statistics
Programming Languages
1 1,531 (59.36%) Python
2 1,272 (49.32%) Java
3 1,044 (40.48%) Scala
4 1,005 (38.97%) SQL
5 529 (20.51%) R
6 259 (10.04%) JavaScript
7 258 (10.00%) C#
8 229 (8.88%) C++
9 157 (6.09%) Java 8
10 129 (5.00%) Perl
11 120 (4.65%) C
12 117 (4.54%) Bash Shell
13 104 (4.03%) MATLAB
14 103 (3.99%) Ruby
15 88 (3.41%) Shell Script
16 65 (2.52%) Go
17 59 (2.29%) T-SQL
18 51 (1.98%) HiveQL
19 38 (1.47%) PL/SQL
20 35 (1.36%) PHP
Qualifications
1 451 (17.49%) Degree
2 170 (6.59%) PhD
3 138 (5.35%) Computer Science Degree
4 96 (3.72%) Security Cleared
5 65 (2.52%) Master's Degree
6 53 (2.06%) DV Cleared
7 49 (1.90%) Numerate Degree
8 21 (0.81%) CEH
8 21 (0.81%) CISSP
9 20 (0.78%) GCIH
9 20 (0.78%) OSCP
10 16 (0.62%) ISTQB
11 14 (0.54%) SC Cleared
12 12 (0.47%) ISTQB Foundation Certification
13 4 (0.16%) AWS Certification
13 4 (0.16%) ISEB
13 4 (0.16%) MBA
13 4 (0.16%) Postgraduate
14 2 (0.078%) MCSA
14 2 (0.078%) MCSD
Quality Assurance & Compliance
1 101 (3.92%) QA
2 63 (2.44%) Data Quality
3 41 (1.59%) GCP
4 14 (0.54%) GDPR
4 14 (0.54%) MiFID
5 9 (0.35%) ISO/IEC 27001
6 8 (0.31%) GPG13
7 6 (0.23%) Solvency II
8 5 (0.19%) SLA
9 3 (0.12%) EMIR
10 2 (0.078%) PMO
11 1 (0.039%) Actionable Recommendations
11 1 (0.039%) Dodd-Frank
11 1 (0.039%) FATCA
11 1 (0.039%) HIPAA
11 1 (0.039%) HL7
11 1 (0.039%) Sarbanes-Oxley
System Software
1 344 (13.34%) Docker
2 337 (13.07%) HDFS
3 195 (7.56%) Apache Flume
4 42 (1.63%) Apache ZooKeeper
5 35 (1.36%) VirtualBox
6 30 (1.16%) VMware Infrastructure
7 24 (0.93%) Virtual Machines
8 20 (0.78%) Active Directory
8 20 (0.78%) Snort
9 13 (0.50%) Hyper-V
10 7 (0.27%) LXC
10 7 (0.27%) Xen
11 5 (0.19%) KVM
12 3 (0.12%) Oracle RAC
13 2 (0.078%) vCloud
13 2 (0.078%) Virtual Servers
13 2 (0.078%) vSphere
14 1 (0.039%) Cronacle
14 1 (0.039%) Samba
Systems Management
1 175 (6.79%) Kubernetes
2 160 (6.20%) Ansible
3 143 (5.54%) Oozie
4 106 (4.11%) Puppet
5 100 (3.88%) Hadoop YARN
6 86 (3.33%) Terraform
7 70 (2.71%) Mesos
8 63 (2.44%) Opscode Chef
9 53 (2.06%) Kibana
10 36 (1.40%) Docker Swarm
11 31 (1.20%) Prometheus
12 26 (1.01%) logstash
13 21 (0.81%) Salt
14 20 (0.78%) Suricata
15 19 (0.74%) Ambari
15 19 (0.74%) Consul
16 18 (0.70%) Grafana
17 17 (0.66%) ITRS Geneos
18 12 (0.47%) Nagios
18 12 (0.47%) ZABBIX
Vendors
1 338 (13.11%) Cloudera
2 244 (9.46%) Google
3 230 (8.92%) Tableau
4 200 (7.75%) Oracle
5 186 (7.21%) Hortonworks
6 167 (6.48%) Talend
7 154 (5.97%) Microsoft
8 140 (5.43%) Informatica
9 85 (3.30%) IBM
10 72 (2.79%) SAP
11 60 (2.33%) Atlassian
12 59 (2.29%) MapR
13 53 (2.06%) Splunk
14 38 (1.47%) Sybase
15 36 (1.40%) VMware
16 35 (1.36%) Alteryx
17 34 (1.32%) Databricks
18 32 (1.24%) EXASOL
19 31 (1.20%) WorldPay
20 30 (1.16%) SnapLogic