Period
to 26 September 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 26 September 2018 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
26 Sep 2018
Same period 2017 Same period 2016
Rank 242 281 375
Rank change year-on-year +39 +94 +297
Permanent jobs citing Apache Spark 2,325 2,185 1,581
As % of all permanent IT jobs advertised in the UK 1.39% 1.23% 0.82%
As % of the Application Platforms category 8.37% 6.88% 4.43%
Number of salaries quoted 1,825 1,730 1,277
UK median annual salary £70,000 £66,000 £65,000
Median salary % change year-on-year +6.06% +1.54% -
10th Percentile £45,000 £41,250 £40,000
90th Percentile £100,000 £100,000 £102,500
UK excluding London median annual salary £60,000 £57,000 £55,000
% change year-on-year +5.26% +3.64% +10.00%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 27,791 31,762 35,678
As % of all permanent IT jobs advertised in the UK 16.60% 17.88% 18.51%
Number of salaries quoted 22,322 25,460 30,261
UK median annual salary £50,000 £47,500 £45,000
Median salary % change year-on-year +5.26% +5.56% -
10th Percentile £28,000 £27,000 £26,750
90th Percentile £82,500 £76,250 £75,000
UK excluding London median annual salary £43,000 £41,000 £40,000
% change year-on-year +4.88% +2.50% -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 26 September 2018.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 26 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +35 2,219 £70,000 +4.48% 239
London +30 1,532 £75,000 +7.14% 150
UK excluding London +52 763 £60,000 +5.26% 112
South East +30 266 £60,000 -7.69% 31
East of England +38 121 £65,000 +30.00% 10
North of England -35 112 £55,000 +4.76% 14
Midlands +71 97 £65,000 +52.94% 18
North West -26 84 £58,750 -2.08% 11
South West +51 82 £60,000 +14.29% 16
Scotland +12 71 £55,000 -21.43% 10
East Midlands +39 68 £75,000 +76.47% 11
West Midlands +36 29 £60,000 +23.71% 7
Yorkshire +16 25 £50,500 -3.81% 2
Wales +4 15 £58,000 +65.71% 9
North East -17 3 £40,000 +14.29% 1

For the 6 months to 26 September 2018, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,662 (71.48%) Hadoop
2 1,584 (68.13%) Big Data
3 1,365 (58.71%) Python
4 1,171 (50.37%) Java
5 919 (39.53%) Scala
6 918 (39.48%) Agile Software Development
7 909 (39.10%) SQL
8 897 (38.58%) Analytics
9 849 (36.52%) Kafka
10 792 (34.06%) Amazon AWS
11 740 (31.83%) Machine Learning
11 740 (31.83%) Data Science
12 712 (30.62%) Apache Hive
13 617 (26.54%) NoSQL
14 438 (18.84%) Finance
14 438 (18.84%) ETL
15 431 (18.54%) R
16 423 (18.19%) Microsoft Azure
17 399 (17.16%) Apache Cassandra
17 399 (17.16%) Degree
18 382 (16.43%) DevOps
19 371 (15.96%) Apache
20 369 (15.87%) Software Engineering
21 361 (15.53%) Linux
22 355 (15.27%) Computer Science
23 335 (14.41%) Artificial Intelligence
23 335 (14.41%) Analytical Skills
24 331 (14.24%) HBase
25 328 (14.11%) Algorithms
26 327 (14.06%) HDFS

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 371 (15.96%) Apache
2 227 (9.76%) Elasticsearch
3 175 (7.53%) Apache Pig
4 111 (4.77%) SAS
5 76 (3.27%) Apache Solr
6 38 (1.63%) Confluence
7 33 (1.42%) Tomcat
8 21 (0.90%) Jupyter
9 18 (0.77%) Apache Airflow
10 14 (0.60%) GlassFish
11 11 (0.47%) Cloud Foundry
11 11 (0.47%) OpenStack
12 10 (0.43%) GemFire
12 10 (0.43%) JBoss
13 8 (0.34%) nginx
14 7 (0.30%) Oracle Workflow
14 7 (0.30%) WebSphere
15 6 (0.26%) IIS
15 6 (0.26%) NServiceBus
15 6 (0.26%) WildFly
Applications
1 22 (0.95%) Microsoft Excel
2 18 (0.77%) Chatbot
3 16 (0.69%) GNU Octave
4 15 (0.65%) SPSS
5 7 (0.30%) Microsoft PowerPoint
6 4 (0.17%) Microsoft Office
6 4 (0.17%) MS Visio
7 3 (0.13%) Microsoft Project
7 3 (0.13%) Weka
8 2 (0.086%) CPLEX
8 2 (0.086%) Mathematica
9 1 (0.043%) Mingle
9 1 (0.043%) SPSS Modeler
9 1 (0.043%) Stata
Business Applications
1 2 (0.086%) NetSuite
2 1 (0.043%) Actimize
2 1 (0.043%) Adobe Campaign
2 1 (0.043%) Eloqua
2 1 (0.043%) Oracle AML
2 1 (0.043%) WebTrends
Cloud Services
1 792 (34.06%) Amazon AWS
2 423 (18.19%) Microsoft Azure
3 213 (9.16%) Google Cloud Platform
4 143 (6.15%) Amazon EMR
5 108 (4.65%) Serverless
6 98 (4.22%) Amazon Kinesis
7 88 (3.78%) Amazon S3
8 73 (3.14%) AWS Lambda
9 61 (2.62%) GitHub
10 55 (2.37%) Cloud Computing
11 52 (2.24%) Amazon EC2
12 44 (1.89%) IBM Cloud
12 44 (1.89%) PaaS
13 40 (1.72%) SaaS
14 27 (1.16%) IaaS
15 23 (0.99%) Azure Data Factory
16 21 (0.90%) Cortana Intelligence Suite
17 18 (0.77%) Amazon SageMaker
17 18 (0.77%) Azure Machine Learning
18 16 (0.69%) AWS CloudFormation
Communications & Networking
1 36 (1.55%) HTTP
2 26 (1.12%) Internet
3 23 (0.99%) Sourcefire
4 20 (0.86%) Kerberos
5 13 (0.56%) Firewall
6 11 (0.47%) DNS
6 11 (0.47%) Network Security
6 11 (0.47%) SMS
7 8 (0.34%) TCP/IP
8 7 (0.30%) SSL
9 5 (0.22%) DAB
10 4 (0.17%) MQTT
11 2 (0.086%) AMQP
11 2 (0.086%) DSL
11 2 (0.086%) Intrusion Detection
11 2 (0.086%) SSH
11 2 (0.086%) VPN
12 1 (0.043%) Intranet
12 1 (0.043%) IPsec
12 1 (0.043%) VLAN
Database & Business Intelligence
1 1,662 (71.48%) Hadoop
2 1,584 (68.13%) Big Data
3 712 (30.62%) Apache Hive
4 617 (26.54%) NoSQL
5 399 (17.16%) Apache Cassandra
6 331 (14.24%) HBase
7 311 (13.38%) MongoDB
8 260 (11.18%) Amazon Redshift
9 236 (10.15%) Data Warehouse
10 231 (9.94%) Impala
11 203 (8.73%) Relational Database
12 174 (7.48%) SQL Server
13 143 (6.15%) Power BI
14 134 (5.76%) Data Lake
15 112 (4.82%) Flink
16 110 (4.73%) PostgreSQL
17 106 (4.56%) Sqoop
18 100 (4.30%) MySQL
19 89 (3.83%) RDBMS
20 77 (3.31%) SQL Server Integration Services
Development Applications
1 226 (9.72%) Git (software)
2 189 (8.13%) Jenkins
3 91 (3.91%) JIRA
4 78 (3.35%) JUnit
5 56 (2.41%) Maven
6 49 (2.11%) Atlassian Bamboo
7 45 (1.94%) Gradle
8 38 (1.63%) Visual Studio
9 35 (1.51%) Selenium
10 34 (1.46%) Vagrant
11 32 (1.38%) Subversion
12 31 (1.33%) TeamCity
13 28 (1.20%) Cucumber
14 26 (1.12%) Bitbucket
15 22 (0.95%) Jasmine
16 21 (0.90%) Mockito
17 17 (0.73%) JMeter
17 17 (0.73%) Visual Studio Team System
18 16 (0.69%) Log4Net
18 16 (0.69%) NUnit
General
1 438 (18.84%) Finance
2 220 (9.46%) Banking
3 179 (7.70%) Retail
4 136 (5.85%) Investment Banking
5 111 (4.77%) Marketing
6 110 (4.73%) Telecoms
7 90 (3.87%) Financial Institution
8 87 (3.74%) Games
9 80 (3.44%) Front Office
10 57 (2.45%) Advertising
11 47 (2.02%) Manufacturing
12 45 (1.94%) Electronics
13 35 (1.51%) Pharmaceutical
14 31 (1.33%) Automotive
14 31 (1.33%) Law
15 13 (0.56%) Publishing
16 8 (0.34%) Military
17 7 (0.30%) Retail Banking
18 6 (0.26%) German Language
18 6 (0.26%) Music Industry
Job Titles
1 608 (26.15%) Developer
2 539 (23.18%) Data Engineer
3 361 (15.53%) Data Scientist
4 323 (13.89%) Big Data Engineer
5 263 (11.31%) Architect
6 242 (10.41%) Java Developer
7 168 (7.23%) Data Architect
8 149 (6.41%) Senior Data Scientist
9 144 (6.19%) Big Data Developer
10 123 (5.29%) Senior Developer
11 107 (4.60%) Scala Developer
12 96 (4.13%) Software Engineer
13 90 (3.87%) Hadoop Engineer
14 85 (3.66%) Hadoop Developer
15 83 (3.57%) Big Data Architect
16 82 (3.53%) Java Engineer
17 74 (3.18%) Consultant
18 73 (3.14%) Senior Data Engineer
19 70 (3.01%) Software Developer
20 68 (2.92%) Python Developer
Libraries, Frameworks & Software Standards
1 849 (36.52%) Kafka
2 212 (9.12%) Web Services
3 172 (7.40%) AngularJS
4 165 (7.10%) Natural Language Processing
5 162 (6.97%) Spring
6 141 (6.06%) HTML
7 140 (6.02%) Node.js
8 136 (5.85%) REST
9 135 (5.81%) .NET
10 131 (5.63%) RESTful
11 126 (5.42%) TensorFlow
12 115 (4.95%) Pandas
13 113 (4.86%) Spark Streaming
14 108 (4.65%) HTML5
15 105 (4.52%) React
16 96 (4.13%) JSON
16 96 (4.13%) scikit-learn
17 85 (3.66%) Akka
18 79 (3.40%) NumPy
18 79 (3.40%) RabbitMQ
Miscellaneous
1 355 (15.27%) Computer Science
2 335 (14.41%) Analytical Skills
3 328 (14.11%) Algorithms
4 137 (5.89%) Data Structures
5 128 (5.51%) Fintech
6 121 (5.20%) Greenfield Project
7 89 (3.83%) Self-Motivation
8 72 (3.10%) Cyberthreat
9 51 (2.19%) Virtual Team
10 48 (2.06%) Derivative
10 48 (2.06%) Management Information System
11 44 (1.89%) User Experience
12 33 (1.42%) Public Cloud
13 31 (1.33%) Linux Command Line
14 30 (1.29%) Mobile App
15 28 (1.20%) Life Science
16 27 (1.16%) Equities
16 27 (1.16%) Social Network
17 24 (1.03%) Blog
18 22 (0.95%) Market Risk
Operating Systems
1 361 (15.53%) Linux
2 175 (7.53%) Unix
3 64 (2.75%) Windows
4 23 (0.99%) Android
5 22 (0.95%) Apple iOS
6 17 (0.73%) Ubuntu
7 9 (0.39%) CentOS
7 9 (0.39%) Red Hat Enterprise Linux
8 6 (0.26%) VMS
9 4 (0.17%) CoreOS
10 2 (0.086%) Windows Server
11 1 (0.043%) AIX
11 1 (0.043%) DC/OS
11 1 (0.043%) Mac OS
11 1 (0.043%) Mesosphere
11 1 (0.043%) Windows Server 2012
Processes & Methodologies
1 918 (39.48%) Agile Software Development
2 897 (38.58%) Analytics
3 740 (31.83%) Data Science
3 740 (31.83%) Machine Learning
4 438 (18.84%) ETL
5 382 (16.43%) DevOps
6 369 (15.87%) Software Engineering
7 335 (14.41%) Artificial Intelligence
8 312 (13.42%) TDD
9 303 (13.03%) Visualisation
10 300 (12.90%) Problem-Solving
11 291 (12.52%) Data Analytics
12 289 (12.43%) Data Modelling
13 275 (11.83%) Mathematics
14 256 (11.01%) Open Source
15 254 (10.92%) Scrum
16 251 (10.80%) Business Intelligence
17 248 (10.67%) Data Engineering
18 232 (9.98%) Microservices
19 212 (9.12%) Continuous Integration
Programming Languages
1 1,365 (58.71%) Python
2 1,171 (50.37%) Java
3 919 (39.53%) Scala
4 909 (39.10%) SQL
5 431 (18.54%) R
6 240 (10.32%) JavaScript
7 235 (10.11%) C#
8 209 (8.99%) C++
9 124 (5.33%) Java 8
10 111 (4.77%) C
11 94 (4.04%) Perl
12 92 (3.96%) Bash Shell
13 87 (3.74%) Shell Script
14 82 (3.53%) Ruby
15 71 (3.05%) MATLAB
16 55 (2.37%) T-SQL
17 49 (2.11%) Go
18 48 (2.06%) PHP
19 34 (1.46%) PowerShell
20 33 (1.42%) TypeScript
Qualifications
1 399 (17.16%) Degree
2 146 (6.28%) PhD
3 135 (5.81%) Computer Science Degree
4 80 (3.44%) Security Cleared
5 53 (2.28%) DV Cleared
6 51 (2.19%) Master's Degree
7 50 (2.15%) Numerate Degree
8 23 (0.99%) CEH
8 23 (0.99%) CISSP
8 23 (0.99%) GCIH
8 23 (0.99%) OSCP
9 17 (0.73%) SC Cleared
10 8 (0.34%) ISTQB
11 5 (0.22%) ISTQB Foundation Certification
12 4 (0.17%) ISEB
13 2 (0.086%) AWS Certification
13 2 (0.086%) MCSA
13 2 (0.086%) MCSD
13 2 (0.086%) Microsoft Certification
14 1 (0.043%) ITIL Foundation Certificate
Quality Assurance & Compliance
1 101 (4.34%) QA
2 58 (2.49%) Data Quality
3 19 (0.82%) GDPR
4 13 (0.56%) ISO/IEC 27001
5 11 (0.47%) GPG13
5 11 (0.47%) MiFID
6 4 (0.17%) SLA
7 3 (0.13%) Solvency II
8 2 (0.086%) PCI DSS
9 1 (0.043%) HL7
9 1 (0.043%) PMO
System Software
1 327 (14.06%) HDFS
2 295 (12.69%) Docker
3 153 (6.58%) Apache Flume
4 44 (1.89%) Apache ZooKeeper
5 37 (1.59%) VirtualBox
6 28 (1.20%) Virtual Machines
7 23 (0.99%) Snort
8 19 (0.82%) Active Directory
9 18 (0.77%) VMware Infrastructure
10 8 (0.34%) Hyper-V
11 3 (0.13%) Oracle RAC
11 3 (0.13%) Xen
12 2 (0.086%) KVM
12 2 (0.086%) LXC
12 2 (0.086%) Virtual Servers
13 1 (0.043%) Firmware
13 1 (0.043%) Samba
13 1 (0.043%) VMware ESXi
Systems Management
1 172 (7.40%) Kubernetes
2 158 (6.80%) Oozie
3 117 (5.03%) Ansible
4 102 (4.39%) Hadoop YARN
5 95 (4.09%) Terraform
6 81 (3.48%) Puppet
7 70 (3.01%) Kibana
8 64 (2.75%) Mesos
9 51 (2.19%) Opscode Chef
10 48 (2.06%) Prometheus
11 37 (1.59%) logstash
12 33 (1.42%) Docker Swarm
12 33 (1.42%) Grafana
13 23 (0.99%) Suricata
14 19 (0.82%) ITRS Geneos
15 13 (0.56%) Ambari
16 11 (0.47%) QRadar
17 10 (0.43%) Consul
17 10 (0.43%) ZABBIX
18 8 (0.34%) Salt
Vendors
1 286 (12.30%) Cloudera
2 240 (10.32%) Google
3 226 (9.72%) Tableau
4 197 (8.47%) Hortonworks
5 171 (7.35%) Oracle
6 159 (6.84%) Microsoft
7 124 (5.33%) Talend
8 104 (4.47%) Informatica
9 94 (4.04%) SAP
10 78 (3.35%) MapR
11 75 (3.23%) IBM
12 70 (3.01%) Databricks
13 55 (2.37%) Splunk
14 40 (1.72%) Atlassian
15 36 (1.55%) Sybase
16 26 (1.12%) Alteryx
17 24 (1.03%) TIBCO
18 23 (0.99%) TippingPoint
19 22 (0.95%) New Relic
20 21 (0.90%) AppDynamics