Period
to 20 April 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 20 April 2018 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
20 Apr 2018
Same period 2017 Same period 2016
Rank 224 295 433
Rank change year-on-year +71 +138 +593
Permanent jobs citing Apache Spark 2,700 1,889 1,491
As % of all permanent IT jobs advertised in the UK 1.52% 1.09% 0.73%
As % of the Application Platforms category 9.12% 6.18% 3.85%
Number of salaries quoted 2,170 1,576 1,256
UK median annual salary £70,000 £66,750 £65,000
Median salary % change year-on-year +4.87% +2.69% -
10th Percentile £42,500 £42,500 £41,000
90th Percentile £103,875 £100,000 £100,000
UK excluding London median annual salary £57,500 £57,000 £57,500
% change year-on-year +0.88% -0.87% +9.52%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 29,613 30,563 38,748
As % of all permanent IT jobs advertised in the UK 16.68% 17.71% 18.93%
Number of salaries quoted 24,265 25,403 32,603
UK median annual salary £50,000 £45,269 £45,000
Median salary % change year-on-year +10.45% +0.60% +2.27%
10th Percentile £27,500 £26,750 £26,500
90th Percentile £80,000 £75,000 £72,500
UK excluding London median annual salary £42,500 £40,000 £40,000
% change year-on-year +6.25% - -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 20 April 2018.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 16 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 20 April 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +72 2,584 £70,000 +6.06% 284
London +70 1,838 £75,000 +7.14% 196
UK excluding London +70 826 £57,500 +0.88% 97
South East +91 227 £60,000 -7.69% 36
North of England +27 183 £55,000 - 16
East of England +65 139 £73,750 +34.09% 13
South West -6 126 £50,000 -9.09% 12
North West +8 122 £54,750 -0.45% 13
Scotland 0 75 £52,500 -32.26% 5
Midlands -38 65 £55,000 +4.76% 11
Yorkshire +47 56 £55,000 - 2
East Midlands -4 38 £55,000 +29.41% 2
West Midlands -38 27 £60,000 - 9
Wales -8 9 £52,700 -8.35% 2
North East -36 5 - - 1
Northern Ireland +4 2 £85,000 +49.12% 2

For the 6 months to 20 April 2018, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,873 (69.37%) Hadoop
2 1,779 (65.89%) Big Data
3 1,625 (60.19%) Python
4 1,278 (47.33%) Java
5 1,100 (40.74%) Scala
6 1,069 (39.59%) Analytics
7 1,023 (37.89%) Agile Software Development
8 990 (36.67%) Machine Learning
9 949 (35.15%) Data Science
10 946 (35.04%) SQL
11 848 (31.41%) Kafka
12 837 (31.00%) Amazon AWS
13 683 (25.30%) Apache Hive
14 654 (24.22%) NoSQL
15 549 (20.33%) Apache Cassandra
16 536 (19.85%) R
17 514 (19.04%) Finance
18 496 (18.37%) Software Engineering
19 478 (17.70%) Linux
20 454 (16.81%) Algorithms
21 453 (16.78%) ETL
22 451 (16.70%) Degree
23 431 (15.96%) DevOps
24 427 (15.81%) Artificial Intelligence
25 426 (15.78%) MongoDB
26 424 (15.70%) HBase
27 402 (14.89%) Docker
28 400 (14.81%) Microsoft Azure
28 400 (14.81%) Analytical Skills
29 374 (13.85%) Computer Science

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 367 (13.59%) Apache
2 340 (12.59%) Elasticsearch
3 161 (5.96%) Apache Pig
4 146 (5.41%) SAS
5 141 (5.22%) Apache Solr
6 33 (1.22%) CMS
7 32 (1.19%) Apache Airflow
8 30 (1.11%) Tomcat
9 27 (1.00%) OpenStack
10 25 (0.93%) Confluence
11 20 (0.74%) JBoss
12 19 (0.70%) nginx
13 16 (0.59%) Luigi
13 16 (0.59%) Oracle SOA Suite
14 13 (0.48%) Jetty
15 12 (0.44%) Jupyter
16 9 (0.33%) WebLogic
17 8 (0.30%) WebSphere
18 6 (0.22%) Oracle Coherence
18 6 (0.22%) SharePoint
Applications
1 43 (1.59%) Microsoft Excel
2 24 (0.89%) SPSS
3 17 (0.63%) Weka
4 12 (0.44%) Chatbot
5 10 (0.37%) GNU Octave
6 8 (0.30%) Mathematica
7 7 (0.26%) Microsoft PowerPoint
8 6 (0.22%) MS Visio
9 1 (0.037%) CPLEX
9 1 (0.037%) HipChat
9 1 (0.037%) Microsoft Office
Business Applications
1 4 (0.15%) Oracle 11i
1 4 (0.15%) Oracle Assets
1 4 (0.15%) Oracle EBS
1 4 (0.15%) Oracle EBS R12
1 4 (0.15%) Oracle General Ledger
1 4 (0.15%) Oracle iProcurement
1 4 (0.15%) Oracle Payables
1 4 (0.15%) Oracle Receivables
2 3 (0.11%) Distributed Ledger
3 1 (0.037%) Actimize
3 1 (0.037%) Dynamics CRM
3 1 (0.037%) Oracle AML
3 1 (0.037%) Summit
3 1 (0.037%) WebTrends
Cloud Services
1 837 (31.00%) Amazon AWS
2 400 (14.81%) Microsoft Azure
3 158 (5.85%) Google Cloud Platform
4 149 (5.52%) Amazon EMR
5 133 (4.93%) Amazon S3
6 117 (4.33%) Amazon Kinesis
7 110 (4.07%) GitHub
8 62 (2.30%) Serverless
9 60 (2.22%) AWS Lambda
9 60 (2.22%) SaaS
10 41 (1.52%) Cloud Computing
11 40 (1.48%) Amazon EC2
12 34 (1.26%) PaaS
13 32 (1.19%) Azure Machine Learning
14 30 (1.11%) IaaS
15 20 (0.74%) IBM Cloud
16 17 (0.63%) Datadog
16 17 (0.63%) OpenShift
17 15 (0.56%) Google Analytics
18 13 (0.48%) AWS CloudFormation
Communications & Networking
1 43 (1.59%) HTTP
2 36 (1.33%) Kerberos
3 30 (1.11%) Internet
4 26 (0.96%) DNS
5 24 (0.89%) TCP/IP
6 23 (0.85%) Firewall
7 22 (0.81%) Sourcefire
8 17 (0.63%) VPN
9 15 (0.56%) Network Security
10 10 (0.37%) SMTP
11 6 (0.22%) DAB
12 5 (0.19%) IPsec
12 5 (0.19%) SMS
13 4 (0.15%) GSM
13 4 (0.15%) ICMP
13 4 (0.15%) LTE
13 4 (0.15%) MQTT
13 4 (0.15%) SSL
13 4 (0.15%) UMTS
14 3 (0.11%) DHCP
Database & Business Intelligence
1 1,873 (69.37%) Hadoop
2 1,779 (65.89%) Big Data
3 683 (25.30%) Apache Hive
4 654 (24.22%) NoSQL
5 549 (20.33%) Apache Cassandra
6 426 (15.78%) MongoDB
7 424 (15.70%) HBase
8 264 (9.78%) Data Warehouse
9 253 (9.37%) Impala
10 250 (9.26%) Relational Database
11 212 (7.85%) Amazon Redshift
12 187 (6.93%) PostgreSQL
13 185 (6.85%) Data Lake
14 174 (6.44%) MySQL
15 170 (6.30%) SQL Server
16 161 (5.96%) Data Mining
17 147 (5.44%) RDBMS
18 112 (4.15%) Power BI
19 109 (4.04%) Flink
20 103 (3.81%) Redis
Development Applications
1 211 (7.81%) Jenkins
2 193 (7.15%) Git (software)
3 140 (5.19%) JIRA
4 76 (2.81%) Maven
5 55 (2.04%) Atlassian Bamboo
6 47 (1.74%) Cucumber
6 47 (1.74%) Selenium
7 39 (1.44%) Gradle
8 37 (1.37%) JUnit
9 34 (1.26%) TeamCity
10 30 (1.11%) Sonatype Nexus
11 27 (1.00%) Vagrant
12 23 (0.85%) RStudio
13 20 (0.74%) Bitbucket
14 18 (0.67%) JMeter
15 16 (0.59%) Subversion
15 16 (0.59%) WATIR
16 15 (0.56%) Artifactory
16 15 (0.56%) GoCD
16 15 (0.56%) Team Foundation Server
General
1 514 (19.04%) Finance
2 168 (6.22%) Retail
3 166 (6.15%) Telecoms
4 154 (5.70%) Marketing
5 146 (5.41%) Banking
6 67 (2.48%) Electronics
7 64 (2.37%) Games
8 59 (2.19%) Manufacturing
9 53 (1.96%) Investment Banking
10 51 (1.89%) Financial Institution
11 50 (1.85%) Advertising
12 49 (1.81%) Front Office
13 35 (1.30%) Pharmaceutical
14 31 (1.15%) Law
15 30 (1.11%) Legal
16 21 (0.78%) Publishing
17 11 (0.41%) Automotive
18 8 (0.30%) Retail Banking
19 7 (0.26%) Military
20 4 (0.15%) Corporate Banking
Job Titles
1 610 (22.59%) Data Engineer
2 552 (20.44%) Developer
3 454 (16.81%) Data Scientist
4 322 (11.93%) Big Data Engineer
5 313 (11.59%) Architect
6 239 (8.85%) Java Developer
7 215 (7.96%) Data Architect
8 158 (5.85%) Software Engineer
9 144 (5.33%) Big Data Architect
10 142 (5.26%) Senior Data Scientist
11 140 (5.19%) Big Data Developer
12 127 (4.70%) Java Engineer
13 116 (4.30%) Python Engineer
13 116 (4.30%) Scala Developer
14 109 (4.04%) Hadoop Engineer
15 100 (3.70%) Senior Developer
16 97 (3.59%) AWS Engineer
17 94 (3.48%) DevOps Engineer
18 92 (3.41%) Machine Learning Engineer
19 88 (3.26%) Senior Data Engineer
Libraries, Frameworks & Software Standards
1 848 (31.41%) Kafka
2 216 (8.00%) Natural Language Processing
3 171 (6.33%) Spring
4 167 (6.19%) Node.js
5 164 (6.07%) Web Services
6 151 (5.59%) HTML
7 150 (5.56%) .NET
8 146 (5.41%) AngularJS
9 134 (4.96%) Pandas
10 129 (4.78%) REST
11 126 (4.67%) Spark Streaming
12 115 (4.26%) CSS
13 107 (3.96%) TensorFlow
14 105 (3.89%) NumPy
14 105 (3.89%) React
15 99 (3.67%) Akka
15 99 (3.67%) scikit-learn
16 92 (3.41%) HTML5
17 91 (3.37%) Hibernate
18 88 (3.26%) Elastic Stack
Miscellaneous
1 454 (16.81%) Algorithms
2 400 (14.81%) Analytical Skills
3 374 (13.85%) Computer Science
4 149 (5.52%) Fintech
5 122 (4.52%) Data Structures
6 114 (4.22%) Greenfield Project
7 78 (2.89%) User Experience
8 62 (2.30%) Self-Motivation
9 46 (1.70%) Clustering
10 43 (1.59%) Cyberthreat
10 43 (1.59%) Linux Command Line
11 40 (1.48%) Hedge funds
12 30 (1.11%) Management Information System
13 29 (1.07%) Mobile App
14 28 (1.04%) Data Centre
14 28 (1.04%) Derivative
15 27 (1.00%) Xbox
16 26 (0.96%) Robotics
16 26 (0.96%) Social Media
17 25 (0.93%) Market Risk
Operating Systems
1 478 (17.70%) Linux
2 163 (6.04%) Unix
3 80 (2.96%) Windows
4 17 (0.63%) Android
4 17 (0.63%) Red Hat Enterprise Linux
5 16 (0.59%) Apple iOS
6 15 (0.56%) Ubuntu
7 12 (0.44%) CentOS
8 8 (0.30%) CoreOS
9 5 (0.19%) Amazon Linux
9 5 (0.19%) Windows Server
10 3 (0.11%) Debian
10 3 (0.11%) Mac OS X
10 3 (0.11%) VMS
10 3 (0.11%) Windows 10
11 2 (0.074%) Mac OS
12 1 (0.037%) AIX
12 1 (0.037%) Mesosphere
12 1 (0.037%) Solaris
12 1 (0.037%) Windows Server 2008
Processes & Methodologies
1 1,069 (39.59%) Analytics
2 1,023 (37.89%) Agile Software Development
3 990 (36.67%) Machine Learning
4 949 (35.15%) Data Science
5 496 (18.37%) Software Engineering
6 453 (16.78%) ETL
7 431 (15.96%) DevOps
8 427 (15.81%) Artificial Intelligence
9 362 (13.41%) TDD
10 342 (12.67%) Visualisation
11 339 (12.56%) Mathematics
12 333 (12.33%) Business Intelligence
13 323 (11.96%) Data Analytics
14 317 (11.74%) Open Source
15 316 (11.70%) Problem-Solving
16 292 (10.81%) Scrum
17 273 (10.11%) Continuous Integration
18 265 (9.81%) Statistics
19 245 (9.07%) Data Modelling
20 236 (8.74%) Data Engineering
Programming Languages
1 1,625 (60.19%) Python
2 1,278 (47.33%) Java
3 1,100 (40.74%) Scala
4 946 (35.04%) SQL
5 536 (19.85%) R
6 294 (10.89%) C#
7 280 (10.37%) JavaScript
8 224 (8.30%) C++
9 160 (5.93%) Java 8
10 142 (5.26%) Bash Shell
11 139 (5.15%) Ruby
12 121 (4.48%) MATLAB
13 110 (4.07%) C
14 105 (3.89%) Go
15 102 (3.78%) Perl
16 81 (3.00%) Shell Script
17 38 (1.41%) T-SQL
18 37 (1.37%) HiveQL
19 35 (1.30%) PL/SQL
20 34 (1.26%) PHP
Qualifications
1 451 (16.70%) Degree
2 197 (7.30%) PhD
3 133 (4.93%) Computer Science Degree
4 96 (3.56%) Security Cleared
5 82 (3.04%) Master's Degree
6 49 (1.81%) DV Cleared
7 45 (1.67%) Numerate Degree
8 23 (0.85%) CEH
8 23 (0.85%) CISSP
9 22 (0.81%) GCIH
9 22 (0.81%) ISTQB
9 22 (0.81%) OSCP
10 16 (0.59%) ISTQB Foundation Certification
11 14 (0.52%) SC Cleared
12 6 (0.22%) MBA
13 4 (0.15%) AWS Certification
13 4 (0.15%) Postgraduate
13 4 (0.15%) PRINCE2 Certification
13 4 (0.15%) TOGAF Certification
14 2 (0.074%) ISEB
Quality Assurance & Compliance
1 92 (3.41%) QA
2 85 (3.15%) Data Quality
3 37 (1.37%) GCP
4 17 (0.63%) ISO/IEC 27001
5 16 (0.59%) GPG13
6 11 (0.41%) GDPR
7 9 (0.33%) Solvency II
8 5 (0.19%) MiFID
9 4 (0.15%) Actionable Recommendations
9 4 (0.15%) EMIR
9 4 (0.15%) SLA
10 2 (0.074%) Dodd-Frank
10 2 (0.074%) FATCA
10 2 (0.074%) PMO
11 1 (0.037%) HIPAA
11 1 (0.037%) Sarbanes-Oxley
System Software
1 402 (14.89%) Docker
2 277 (10.26%) HDFS
3 187 (6.93%) Apache Flume
4 46 (1.70%) Apache ZooKeeper
5 31 (1.15%) VMware Infrastructure
6 22 (0.81%) Snort
7 21 (0.78%) VirtualBox
8 20 (0.74%) Virtual Machines
9 16 (0.59%) Hyper-V
10 15 (0.56%) Active Directory
11 6 (0.22%) LXC
12 5 (0.19%) Xen
13 3 (0.11%) KVM
14 2 (0.074%) vCloud
14 2 (0.074%) vSphere
15 1 (0.037%) Cronacle
15 1 (0.037%) GridGain
15 1 (0.037%) MapR-FS
15 1 (0.037%) Virtual Servers
Systems Management
1 190 (7.04%) Kubernetes
2 180 (6.67%) Ansible
3 157 (5.81%) Hadoop YARN
4 127 (4.70%) Puppet
5 113 (4.19%) Oozie
6 107 (3.96%) Terraform
7 84 (3.11%) Mesos
8 77 (2.85%) Opscode Chef
9 32 (1.19%) Kibana
10 24 (0.89%) Salt
11 23 (0.85%) Docker Swarm
12 22 (0.81%) Suricata
13 20 (0.74%) Ambari
14 16 (0.59%) logstash
15 14 (0.52%) Nagios
16 13 (0.48%) Consul
16 13 (0.48%) Prometheus
17 11 (0.41%) QRadar
18 8 (0.30%) ZABBIX
19 7 (0.26%) ITRS Geneos
Vendors
1 318 (11.78%) Cloudera
2 272 (10.07%) Google
3 256 (9.48%) Tableau
4 215 (7.96%) Oracle
5 183 (6.78%) Microsoft
6 159 (5.89%) Hortonworks
7 128 (4.74%) Talend
8 114 (4.22%) Informatica
9 91 (3.37%) SAP
10 85 (3.15%) IBM
11 65 (2.41%) Teradata
12 63 (2.33%) MapR
13 54 (2.00%) Atlassian
14 53 (1.96%) Alteryx
15 48 (1.78%) VMware
16 43 (1.59%) Twitter
17 42 (1.56%) Splunk
18 30 (1.11%) Greenplum
19 28 (1.04%) WorldPay
20 25 (0.93%) TIBCO