Period
to 17 November 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 17 November 2018 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
17 Nov 2018
Same period 2017 Same period 2016
Rank 249 271 359
Rank change year-on-year +22 +88 +267
Permanent jobs citing Apache Spark 2,236 2,334 1,654
As % of all permanent IT jobs advertised in the UK 1.38% 1.29% 0.87%
As % of the Application Platforms category 8.52% 7.36% 4.75%
Number of salaries quoted 1,733 1,872 1,372
UK median annual salary £70,000 £65,000 £67,500
Median salary % change year-on-year +7.69% -3.70% +3.85%
10th Percentile £43,850 £41,250 £40,000
90th Percentile £102,500 £100,000 £100,000
UK excluding London median annual salary £62,138 £52,700 £55,000
% change year-on-year +17.91% -4.18% -

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 26,233 31,699 34,824
As % of all permanent IT jobs advertised in the UK 16.20% 17.58% 18.31%
Number of salaries quoted 21,105 25,625 29,400
UK median annual salary £50,000 £47,500 £45,000
Median salary % change year-on-year +5.26% +5.56% -
10th Percentile £27,750 £27,000 £26,750
90th Percentile £82,500 £77,500 £75,000
UK excluding London median annual salary £43,500 £40,000 £40,000
% change year-on-year +8.75% - -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 17 November 2018.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 17 November 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +23 2,124 £72,500 +8.21% 311
London -5 1,396 £78,750 +12.50% 192
UK excluding London +68 820 £62,138 +17.91% 133
South East +33 263 £60,000 - 41
North of England +39 147 £70,000 +40.00% 32
South West +44 121 £60,000 +20.00% 24
Midlands +86 108 £75,000 +76.47% 13
North West +11 104 £70,000 +40.00% 17
East of England +6 83 £66,000 +10.00% 9
East Midlands +73 80 £75,000 +76.47% 8
Scotland +21 77 £55,000 -15.38% 5
Yorkshire +44 39 £50,500 +3.59% 10
West Midlands +42 28 £52,500 +23.53% 5
Wales -3 19 £54,000 +48.46% 4
North East - 8 £100,000 - 5
Northern Ireland 0 2 £42,500 - 5
Channel Islands - 1 £37,500 -

For the 6 months to 17 November 2018, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,635 (73.12%) Hadoop
2 1,521 (68.02%) Big Data
3 1,358 (60.73%) Python
4 1,153 (51.57%) Java
5 954 (42.67%) Agile Software Development
6 900 (40.25%) Scala
7 891 (39.85%) Kafka
8 866 (38.73%) SQL
9 813 (36.36%) Analytics
10 809 (36.18%) Amazon AWS
11 746 (33.36%) Apache Hive
12 723 (32.33%) Machine Learning
13 720 (32.20%) Data Science
14 652 (29.16%) NoSQL
15 466 (20.84%) Finance
16 449 (20.08%) Microsoft Azure
17 443 (19.81%) Apache Cassandra
18 442 (19.77%) Software Engineering
19 421 (18.83%) Degree
20 415 (18.56%) Apache
21 401 (17.93%) R
21 401 (17.93%) DevOps
22 394 (17.62%) ETL
23 385 (17.22%) HBase
24 379 (16.95%) Data Modelling
25 360 (16.10%) Computer Science
26 349 (15.61%) Artificial Intelligence
27 342 (15.30%) Linux
28 322 (14.40%) MongoDB
29 305 (13.64%) Data Analytics

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 415 (18.56%) Apache
2 283 (12.66%) Elasticsearch
3 200 (8.94%) Apache Pig
4 125 (5.59%) SAS
5 88 (3.94%) Apache Solr
6 30 (1.34%) Tomcat
7 29 (1.30%) Confluence
8 22 (0.98%) Jupyter
9 21 (0.94%) GlassFish
10 18 (0.81%) Apache Airflow
11 17 (0.76%) nginx
12 14 (0.63%) JBoss
13 12 (0.54%) WebSphere
14 8 (0.36%) Cloud Foundry
15 7 (0.31%) Apache Geronimo
15 7 (0.31%) GemFire
15 7 (0.31%) OpenStack
16 6 (0.27%) IIS
16 6 (0.27%) WildFly
17 5 (0.22%) Oracle Workflow
Applications
1 18 (0.81%) Chatbot
2 17 (0.76%) SPSS
3 12 (0.54%) Microsoft Excel
4 10 (0.45%) GNU Octave
5 7 (0.31%) Weka
6 6 (0.27%) Microsoft Office
6 6 (0.27%) Stata
7 5 (0.22%) Microsoft PowerPoint
8 3 (0.13%) MS Visio
9 1 (0.045%) CPLEX
9 1 (0.045%) Microsoft Project
Business Applications
1 2 (0.089%) NetSuite
2 1 (0.045%) Adobe Campaign
2 1 (0.045%) Eloqua
2 1 (0.045%) WebTrends
Cloud Services
1 809 (36.18%) Amazon AWS
2 449 (20.08%) Microsoft Azure
3 251 (11.23%) Google Cloud Platform
4 130 (5.81%) Amazon EMR
5 106 (4.74%) Serverless
6 97 (4.34%) Cloud Computing
7 76 (3.40%) GitHub
8 72 (3.22%) AWS Lambda
9 69 (3.09%) Amazon Kinesis
10 67 (3.00%) Amazon EC2
11 52 (2.33%) Amazon S3
12 46 (2.06%) PaaS
13 38 (1.70%) SaaS
14 34 (1.52%) AWS CloudFormation
15 33 (1.48%) Azure Data Factory
16 28 (1.25%) IBM Cloud
17 26 (1.16%) Cortana Intelligence Suite
17 26 (1.16%) IaaS
18 24 (1.07%) Amazon SageMaker
19 23 (1.03%) Slack
Communications & Networking
1 53 (2.37%) HTTP
2 45 (2.01%) Internet
3 18 (0.81%) Kerberos
4 14 (0.63%) DNS
5 12 (0.54%) Firewall
6 10 (0.45%) Sourcefire
7 8 (0.36%) AMQP
7 8 (0.36%) TCP/IP
8 7 (0.31%) DAB
8 7 (0.31%) SSL
9 5 (0.22%) SMS
10 4 (0.18%) Network Security
11 3 (0.13%) SSH
12 2 (0.089%) DSL
12 2 (0.089%) Intrusion Detection
12 2 (0.089%) VPN
13 1 (0.045%) Intranet
13 1 (0.045%) IPsec
13 1 (0.045%) WAN
13 1 (0.045%) Wi-Fi
Database & Business Intelligence
1 1,635 (73.12%) Hadoop
2 1,521 (68.02%) Big Data
3 746 (33.36%) Apache Hive
4 652 (29.16%) NoSQL
5 443 (19.81%) Apache Cassandra
6 385 (17.22%) HBase
7 322 (14.40%) MongoDB
8 256 (11.45%) Amazon Redshift
9 240 (10.73%) Data Warehouse
10 237 (10.60%) Impala
11 185 (8.27%) SQL Server
12 178 (7.96%) Relational Database
13 146 (6.53%) Data Lake
13 146 (6.53%) Flink
14 143 (6.40%) Power BI
15 131 (5.86%) PostgreSQL
16 105 (4.70%) MySQL
17 93 (4.16%) Sqoop
18 80 (3.58%) RDBMS
18 80 (3.58%) SQL Server Integration Services
Development Applications
1 229 (10.24%) Git (software)
2 191 (8.54%) Jenkins
3 94 (4.20%) JIRA
4 54 (2.42%) JUnit
5 51 (2.28%) Visual Studio
6 49 (2.19%) Gradle
7 42 (1.88%) Maven
8 41 (1.83%) Atlassian Bamboo
8 41 (1.83%) Selenium
9 38 (1.70%) Subversion
10 27 (1.21%) TeamCity
11 25 (1.12%) GoCD
11 25 (1.12%) JMeter
12 24 (1.07%) CircleCI
12 24 (1.07%) Vagrant
12 24 (1.07%) Visual Studio Team System
13 21 (0.94%) SoapUI
14 20 (0.89%) Bitbucket
15 19 (0.85%) GitLab
15 19 (0.85%) Jasmine
General
1 466 (20.84%) Finance
2 203 (9.08%) Retail
3 171 (7.65%) Banking
4 150 (6.71%) Telecoms
5 112 (5.01%) Marketing
6 93 (4.16%) Investment Banking
7 91 (4.07%) Law
8 86 (3.85%) Games
9 51 (2.28%) Military
10 50 (2.24%) Manufacturing
11 48 (2.15%) Financial Institution
12 46 (2.06%) Electronics
13 44 (1.97%) Advertising
14 39 (1.74%) Automotive
14 39 (1.74%) Front Office
15 23 (1.03%) Pharmaceutical
16 12 (0.54%) Retail Banking
17 8 (0.36%) Billing
18 6 (0.27%) German Language
18 6 (0.27%) Music Industry
Job Titles
1 597 (26.70%) Developer
2 508 (22.72%) Data Engineer
3 330 (14.76%) Data Scientist
4 324 (14.49%) Big Data Engineer
5 265 (11.85%) Architect
6 212 (9.48%) Java Developer
7 166 (7.42%) Data Architect
8 144 (6.44%) Big Data Developer
9 139 (6.22%) Senior Data Scientist
10 124 (5.55%) Senior Developer
11 110 (4.92%) Software Engineer
12 94 (4.20%) Consultant
13 92 (4.11%) Scala Developer
14 87 (3.89%) Senior Data Engineer
15 85 (3.80%) Hadoop Engineer
15 85 (3.80%) Python Developer
16 79 (3.53%) Java Engineer
16 79 (3.53%) Software Developer
17 75 (3.35%) Big Data Architect
18 73 (3.26%) Hadoop Developer
Libraries, Frameworks & Software Standards
1 891 (39.85%) Kafka
2 233 (10.42%) Web Services
3 181 (8.09%) AngularJS
4 165 (7.38%) Node.js
5 154 (6.89%) HTML
6 152 (6.80%) Natural Language Processing
7 136 (6.08%) Spring
8 135 (6.04%) RESTful
9 134 (5.99%) Spark Streaming
10 129 (5.77%) REST
11 127 (5.68%) TensorFlow
12 119 (5.32%) React
13 116 (5.19%) HTML5
14 111 (4.96%) .NET
15 99 (4.43%) Pandas
15 99 (4.43%) scikit-learn
16 95 (4.25%) JSON
17 91 (4.07%) Akka
18 86 (3.85%) D3.js
19 85 (3.80%) CSS
Miscellaneous
1 360 (16.10%) Computer Science
2 291 (13.01%) Algorithms
3 290 (12.97%) Analytical Skills
4 121 (5.41%) Fintech
5 114 (5.10%) Data Structures
6 101 (4.52%) Greenfield Project
7 84 (3.76%) Self-Motivation
8 46 (2.06%) Derivative
9 45 (2.01%) Virtual Team
10 42 (1.88%) Public Cloud
11 41 (1.83%) Social Network
12 40 (1.79%) User Experience
13 34 (1.52%) Cyberthreat
14 30 (1.34%) Blog
14 30 (1.34%) Management Information System
15 29 (1.30%) Life Science
15 29 (1.30%) Mobile App
16 28 (1.25%) Linux Command Line
17 27 (1.21%) Market Risk
18 22 (0.98%) Social Media
Operating Systems
1 342 (15.30%) Linux
2 185 (8.27%) Unix
3 66 (2.95%) Windows
4 22 (0.98%) Android
4 22 (0.98%) Apple iOS
5 16 (0.72%) Ubuntu
6 13 (0.58%) CentOS
7 11 (0.49%) Red Hat Enterprise Linux
8 6 (0.27%) VMS
9 3 (0.13%) CoreOS
10 2 (0.089%) DC/OS
10 2 (0.089%) Windows Server
11 1 (0.045%) AIX
11 1 (0.045%) Mac OS
11 1 (0.045%) Mesosphere
11 1 (0.045%) Windows Server 2012
Processes & Methodologies
1 954 (42.67%) Agile Software Development
2 813 (36.36%) Analytics
3 723 (32.33%) Machine Learning
4 720 (32.20%) Data Science
5 442 (19.77%) Software Engineering
6 401 (17.93%) DevOps
7 394 (17.62%) ETL
8 379 (16.95%) Data Modelling
9 349 (15.61%) Artificial Intelligence
10 305 (13.64%) Data Analytics
10 305 (13.64%) Visualisation
11 300 (13.42%) Open Source
12 298 (13.33%) TDD
13 282 (12.61%) Data Engineering
14 275 (12.30%) Business Intelligence
14 275 (12.30%) Mathematics
15 246 (11.00%) Microservices
16 243 (10.87%) Scrum
17 220 (9.84%) Problem-Solving
18 205 (9.17%) Statistics
Programming Languages
1 1,358 (60.73%) Python
2 1,153 (51.57%) Java
3 900 (40.25%) Scala
4 866 (38.73%) SQL
5 401 (17.93%) R
6 242 (10.82%) JavaScript
7 232 (10.38%) C#
8 189 (8.45%) C++
9 92 (4.11%) Java 8
10 89 (3.98%) C
11 85 (3.80%) Ruby
12 83 (3.71%) Bash Shell
13 76 (3.40%) Shell Script
14 64 (2.86%) Perl
15 59 (2.64%) Go
16 56 (2.50%) T-SQL
17 53 (2.37%) PHP
18 51 (2.28%) MATLAB
19 40 (1.79%) PowerShell
20 30 (1.34%) TypeScript
Qualifications
1 421 (18.83%) Degree
2 149 (6.66%) Computer Science Degree
3 114 (5.10%) PhD
4 76 (3.40%) Security Cleared
5 62 (2.77%) Numerate Degree
6 41 (1.83%) DV Cleared
7 37 (1.65%) Master's Degree
8 30 (1.34%) SC Cleared
9 10 (0.45%) CEH
9 10 (0.45%) CISSP
9 10 (0.45%) GCIH
9 10 (0.45%) OSCP
10 7 (0.31%) ISEB
11 6 (0.27%) ISTQB
12 4 (0.18%) MCAD
12 4 (0.18%) MCDBA
12 4 (0.18%) MCSD
12 4 (0.18%) Microsoft Certification
13 2 (0.089%) AWS Certification
13 2 (0.089%) ISTQB Foundation Certification
Quality Assurance & Compliance
1 100 (4.47%) QA
2 76 (3.40%) Data Quality
3 19 (0.85%) GDPR
4 16 (0.72%) MiFID
5 6 (0.27%) ISO/IEC 27001
6 4 (0.18%) PCI DSS
7 3 (0.13%) GPG13
7 3 (0.13%) Solvency II
8 2 (0.089%) SLA
9 1 (0.045%) EMIR
9 1 (0.045%) HL7
9 1 (0.045%) ISO 9001
9 1 (0.045%) PMO
System Software
1 289 (12.92%) Docker
2 281 (12.57%) HDFS
3 145 (6.48%) Apache Flume
4 56 (2.50%) Apache ZooKeeper
5 27 (1.21%) Virtual Machines
6 26 (1.16%) VirtualBox
7 18 (0.81%) Active Directory
8 12 (0.54%) VMware Infrastructure
9 10 (0.45%) Snort
10 5 (0.22%) Hyper-V
11 3 (0.13%) Xen
12 2 (0.089%) KVM
13 1 (0.045%) Firmware
13 1 (0.045%) Oracle RAC
13 1 (0.045%) Platform Symphony
13 1 (0.045%) VMware ESXi
Systems Management
1 190 (8.50%) Oozie
2 171 (7.65%) Kubernetes
3 149 (6.66%) Hadoop YARN
4 124 (5.55%) Ansible
5 112 (5.01%) Terraform
6 89 (3.98%) Kibana
7 87 (3.89%) Puppet
8 70 (3.13%) Mesos
9 59 (2.64%) Opscode Chef
10 51 (2.28%) logstash
11 46 (2.06%) Prometheus
12 39 (1.74%) Grafana
13 23 (1.03%) Ambari
14 22 (0.98%) Docker Swarm
15 13 (0.58%) ITRS Geneos
16 10 (0.45%) Suricata
17 9 (0.40%) ZABBIX
18 8 (0.36%) Nagios
19 7 (0.31%) Salt
20 6 (0.27%) Xymon
Vendors
1 283 (12.66%) Cloudera
2 239 (10.69%) Tableau
3 238 (10.64%) Hortonworks
4 232 (10.38%) Google
5 182 (8.14%) Microsoft
6 167 (7.47%) Oracle
7 111 (4.96%) SAP
8 101 (4.52%) MapR
9 97 (4.34%) Databricks
10 89 (3.98%) Talend
11 88 (3.94%) Informatica
12 66 (2.95%) IBM
13 55 (2.46%) Splunk
14 33 (1.48%) CA Agile Central
15 30 (1.34%) Alteryx
16 29 (1.30%) TIBCO
17 24 (1.07%) Teradata
18 23 (1.03%) Ab Initio
19 22 (0.98%) Sybase
20 21 (0.94%) Atlassian