Period
to 23 May 2019

The following table provides summary statistics for permanent job vacancies with a requirement for Apache Spark skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Apache Spark over the 6 months to 23 May 2019 with a comparison to the same period in the previous 2 years.

Apache Spark
UK
6 months to
23 May 2019
Same period 2018 Same period 2017
Rank 227 228 299
Rank change year-on-year +1 +71 +100
Permanent jobs citing Apache Spark 2,151 2,640 1,898
As % of all permanent IT jobs advertised in the UK 1.49% 1.54% 1.12%
As % of the Application Platforms category 9.73% 9.13% 6.29%
Number of salaries quoted 1,696 2,080 1,569
UK median annual salary £75,000 £70,000 £67,500
Median salary % change year-on-year +7.14% +3.70% +3.85%
10th Percentile £47,500 £43,750 £42,500
90th Percentile £107,500 £103,750 £100,000
UK excluding London median annual salary £60,000 £60,000 £57,000
% change year-on-year - +5.26% -0.87%

Apache Spark is in the Application Platforms category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for application platform skills.

All Application Platform Skills
UK
Permanent vacancies with a requirement for application platform skills 22,111 28,904 30,172
As % of all permanent IT jobs advertised in the UK 15.35% 16.82% 17.83%
Number of salaries quoted 17,296 23,441 24,964
UK median annual salary £50,000 £50,000 £45,500
Median salary % change year-on-year - +9.89% +1.11%
10th Percentile £28,188 £27,951 £26,750
90th Percentile £85,000 £80,000 £75,000
UK excluding London median annual salary £43,000 £42,500 £40,000
% change year-on-year +1.18% +6.25% -

Apache Spark
Job Vacancy Trend

Job postings citing Apache Spark as a percentage of all IT jobs advertised.

Job vacancy trend for Apache Spark in the UK

Apache Spark
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Apache Spark.

Salary trend for Apache Spark in the UK

Apache Spark
Salary Histogram

The salary distribution of IT jobs citing Apache Spark over the 6 months to 23 May 2019.

Salary histogram for Apache Spark in the UK

Apache Spark
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Apache Spark within the UK over the 6 months to 23 May 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -2 2,053 £75,000 +7.14% 96
London +2 1,502 £80,000 +6.67% 57
UK excluding London -20 610 £60,000 - 48
South East -28 146 £70,750 +17.92% 7
North of England -10 131 £55,000 - 12
South West +48 128 £55,000 +10.00% 7
Midlands +48 89 £65,000 +8.33% 9
North West +6 80 £65,000 +18.18% 9
Scotland +6 65 £52,500 -4.55% 9
East Midlands +42 59 £57,500 -4.17% 3
East of England -30 46 £62,500 -13.79% 4
Yorkshire -6 36 £49,337 -10.30% 3
West Midlands +19 30 £70,000 +16.67% 6
North East +25 15 £49,337 -
Northern Ireland -23 4 £42,500 -50.00%
Wales +16 2 £32,250 -32.11%
Channel Islands - 1 £37,500 -

For the 6 months to 23 May 2019, IT jobs citing Apache Spark also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Apache Spark.

1 1,523 (70.80%) Big Data
2 1,481 (68.85%) Hadoop
3 1,463 (68.01%) Python
4 1,200 (55.79%) Java
5 976 (45.37%) Agile Software Development
6 959 (44.58%) Scala
7 943 (43.84%) Kafka
8 934 (43.42%) Amazon AWS
9 906 (42.12%) Analytics
10 879 (40.86%) SQL
11 737 (34.26%) Machine Learning
12 734 (34.12%) Apache Hive
13 721 (33.52%) Data Science
14 617 (28.68%) NoSQL
15 600 (27.89%) DevOps
16 517 (24.04%) Microsoft Azure
17 500 (23.25%) Software Engineering
18 454 (21.11%) ETL
19 427 (19.85%) Google Cloud Platform
20 414 (19.25%) HBase
21 411 (19.11%) Apache Cassandra
22 402 (18.69%) Data Modelling
23 397 (18.46%) Apache
24 396 (18.41%) Finance
25 391 (18.18%) Docker
26 386 (17.95%) Linux
27 384 (17.85%) Data Engineering
28 373 (17.34%) Artificial Intelligence
29 355 (16.50%) Cloudera
30 342 (15.90%) Data Analytics

Apache Spark
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 397 (18.46%) Apache
2 308 (14.32%) Elasticsearch
3 300 (13.95%) Apache Pig
4 110 (5.11%) Apache Solr
5 69 (3.21%) SAS
6 36 (1.67%) Apache Airflow
7 21 (0.98%) Confluence
7 21 (0.98%) Jupyter
8 16 (0.74%) Tomcat
9 14 (0.65%) GlassFish
9 14 (0.65%) WebSphere
10 11 (0.51%) JBoss
10 11 (0.51%) Luigi
10 11 (0.51%) nginx
11 9 (0.42%) IPython
12 8 (0.37%) OpenStack
13 7 (0.33%) Apache Geronimo
13 7 (0.33%) Drupal
14 5 (0.23%) CMS
14 5 (0.23%) Hyperledger
Applications
1 20 (0.93%) Microsoft Excel
2 9 (0.42%) Weka
3 8 (0.37%) Microsoft Office
4 7 (0.33%) Microsoft PowerPoint
4 7 (0.33%) SPSS
5 5 (0.23%) MS Visio
6 2 (0.093%) CPLEX
6 2 (0.093%) Gephi
6 2 (0.093%) GNU Octave
6 2 (0.093%) S-PLUS
6 2 (0.093%) Stata
7 1 (0.046%) Chatbot
Business Applications
1 2 (0.093%) PolarLake
1 2 (0.093%) ReMetrica
1 2 (0.093%) Towers Watson Igloo
2 1 (0.046%) Adobe Campaign
2 1 (0.046%) SAP Oil and Gas
Cloud Services
1 934 (43.42%) Amazon AWS
2 517 (24.04%) Microsoft Azure
3 427 (19.85%) Google Cloud Platform
4 112 (5.21%) Power Platform
5 111 (5.16%) Amazon EMR
6 104 (4.83%) Cloud Computing
7 96 (4.46%) Amazon S3
8 90 (4.18%) Serverless
9 77 (3.58%) Amazon EC2
10 76 (3.53%) GitHub
11 71 (3.30%) Amazon Kinesis
12 65 (3.02%) AWS Lambda
13 58 (2.70%) Azure Data Factory
13 58 (2.70%) PaaS
14 52 (2.42%) SaaS
15 34 (1.58%) Amazon SQS
16 28 (1.30%) Cortana Intelligence Suite
17 20 (0.93%) AWS CloudFormation
18 18 (0.84%) Amazon ELB
19 17 (0.79%) Azure Functions
Communications & Networking
1 41 (1.91%) Internet
2 25 (1.16%) HTTP
3 13 (0.60%) AMQP
3 13 (0.60%) TCP/IP
4 12 (0.56%) Wi-Fi
5 10 (0.46%) Firewall
5 10 (0.46%) LAN
5 10 (0.46%) Software-Defined Networking
5 10 (0.46%) WAN
6 8 (0.37%) Broadband
6 8 (0.37%) VPN
7 7 (0.33%) DNS
8 6 (0.28%) HTTPS
8 6 (0.28%) Kerberos
8 6 (0.28%) MQTT
9 5 (0.23%) SSL
10 4 (0.19%) FTP
11 2 (0.093%) MPLS
11 2 (0.093%) SMTP
11 2 (0.093%) Unified Communications
Database & Business Intelligence
1 1,523 (70.80%) Big Data
2 1,481 (68.85%) Hadoop
3 734 (34.12%) Apache Hive
4 617 (28.68%) NoSQL
5 414 (19.25%) HBase
6 411 (19.11%) Apache Cassandra
7 310 (14.41%) MongoDB
8 297 (13.81%) Impala
9 276 (12.83%) Data Warehouse
10 275 (12.78%) Amazon Redshift
11 184 (8.55%) PostgreSQL
12 181 (8.41%) Sqoop
13 173 (8.04%) Data Lake
14 143 (6.65%) MySQL
15 142 (6.60%) Flink
16 129 (6.00%) SQL Server
17 116 (5.39%) Relational Database
18 112 (5.21%) Power BI
19 106 (4.93%) Neo4j
20 89 (4.14%) Data Mining
Development Applications
1 197 (9.16%) Git (software)
2 192 (8.93%) Jenkins
3 81 (3.77%) JIRA
4 62 (2.88%) Maven
5 44 (2.05%) Gradle
6 42 (1.95%) Visual Studio
7 35 (1.63%) Ant
8 34 (1.58%) Atlassian Bamboo
9 33 (1.53%) Artifactory
10 29 (1.35%) Selenium
11 23 (1.07%) Mercurial
12 21 (0.98%) Visual Studio Team System
13 20 (0.93%) Bitbucket Server
13 20 (0.93%) Travis CI
14 19 (0.88%) Bitbucket
14 19 (0.88%) Cucumber
14 19 (0.88%) Subversion
15 16 (0.74%) GoCD
15 16 (0.74%) JUnit
16 14 (0.65%) IntelliJ
General
1 396 (18.41%) Finance
2 236 (10.97%) Law
3 173 (8.04%) Banking
4 140 (6.51%) Retail
5 86 (4.00%) Telecoms
6 66 (3.07%) Marketing
7 39 (1.81%) Publishing
8 33 (1.53%) Investment Banking
9 32 (1.49%) Games
10 30 (1.39%) Advertising
11 23 (1.07%) Manufacturing
12 20 (0.93%) Financial Institution
13 15 (0.70%) Electronics
14 14 (0.65%) Front Office
15 13 (0.60%) Automotive
16 10 (0.46%) Retail Banking
17 9 (0.42%) Billing
17 9 (0.42%) Legal
18 8 (0.37%) Pharmaceutical
19 6 (0.28%) Aerospace
Job Titles
1 759 (35.29%) Data Engineer
2 409 (19.01%) Developer
3 391 (18.18%) Big Data Engineer
4 217 (10.09%) Java Developer
5 213 (9.90%) Data Scientist
6 207 (9.62%) Architect
7 180 (8.37%) Senior Data Engineer
8 167 (7.76%) Data Architect
9 162 (7.53%) Python Engineer
10 131 (6.09%) Hadoop Engineer
11 122 (5.67%) Software Engineer
12 110 (5.11%) Big Data Developer
13 96 (4.46%) Big Data Architect
14 93 (4.32%) DevOps Engineer
15 90 (4.18%) Java Engineer
16 82 (3.81%) Senior Developer
17 74 (3.44%) AWS Engineer
18 71 (3.30%) Senior Data Scientist
19 65 (3.02%) Scala Developer
20 59 (2.74%) Python Developer
Libraries, Frameworks & Software Standards
1 943 (43.84%) Kafka
2 201 (9.34%) Web Services
3 147 (6.83%) Spark Streaming
4 143 (6.65%) Apache NiFi
5 127 (5.90%) Spring
6 126 (5.86%) Natural Language Processing
7 115 (5.35%) REST
8 113 (5.25%) Pandas
8 113 (5.25%) scikit-learn
9 110 (5.11%) RESTful
10 105 (4.88%) Node.js
11 92 (4.28%) TensorFlow
12 86 (4.00%) .NET
13 80 (3.72%) NumPy
14 75 (3.49%) JSON
14 75 (3.49%) React
15 74 (3.44%) AngularJS
16 72 (3.35%) Elastic Stack
17 70 (3.25%) Akka
18 63 (2.93%) PySpark
Miscellaneous
1 313 (14.55%) Analytical Skills
2 261 (12.13%) Algorithms
3 165 (7.67%) Data Structures
4 93 (4.32%) Fintech
5 67 (3.11%) Greenfield Project
6 54 (2.51%) Management Information System
7 47 (2.19%) Virtual Team
8 42 (1.95%) Self-Motivation
9 39 (1.81%) User Experience
10 37 (1.72%) Social Media
11 30 (1.39%) Public Cloud
12 27 (1.26%) Cloud Native
13 24 (1.12%) Life Science
14 23 (1.07%) Enterprise Software
14 23 (1.07%) Mobile App
15 22 (1.02%) Robotics
16 21 (0.98%) Credit Risk
17 19 (0.88%) Analytical Mindset
18 15 (0.70%) Video on Demand
19 13 (0.60%) Blog
Operating Systems
1 386 (17.95%) Linux
2 189 (8.79%) Unix
3 59 (2.74%) Windows
4 19 (0.88%) CentOS
5 18 (0.84%) Android
5 18 (0.84%) Apple iOS
6 17 (0.79%) Ubuntu
7 12 (0.56%) Windows Server
8 4 (0.19%) Mac OS X
8 4 (0.19%) Oracle Linux
9 2 (0.093%) Debian
10 1 (0.046%) Red Hat Enterprise Linux
Processes & Methodologies
1 976 (45.37%) Agile Software Development
2 906 (42.12%) Analytics
3 737 (34.26%) Machine Learning
4 721 (33.52%) Data Science
5 600 (27.89%) DevOps
6 500 (23.25%) Software Engineering
7 454 (21.11%) ETL
8 402 (18.69%) Data Modelling
9 384 (17.85%) Data Engineering
10 373 (17.34%) Artificial Intelligence
11 342 (15.90%) Data Analytics
12 330 (15.34%) Business Intelligence
13 297 (13.81%) Visualisation
14 271 (12.60%) MapReduce
15 243 (11.30%) Open Source
16 232 (10.79%) Data Analysis
17 227 (10.55%) Continuous Integration
18 225 (10.46%) Microservices
19 224 (10.41%) Scrum
20 222 (10.32%) Computer Science
Programming Languages
1 1,463 (68.01%) Python
2 1,200 (55.79%) Java
3 959 (44.58%) Scala
4 879 (40.86%) SQL
5 244 (11.34%) R
6 157 (7.30%) JavaScript
7 128 (5.95%) C#
8 98 (4.56%) C++
9 76 (3.53%) Bash Shell
10 75 (3.49%) C
11 66 (3.07%) Shell Script
12 64 (2.98%) Java 8
13 58 (2.70%) PL/SQL
14 54 (2.51%) Perl
15 46 (2.14%) Ruby
16 44 (2.05%) T-SQL
17 35 (1.63%) Groovy
18 25 (1.16%) PHP
19 23 (1.07%) Clojure
20 22 (1.02%) TypeScript
Qualifications
1 313 (14.55%) Degree
2 116 (5.39%) Computer Science Degree
3 86 (4.00%) PhD
3 86 (4.00%) Security Cleared
4 60 (2.79%) Numerate Degree
5 56 (2.60%) SC Cleared
6 32 (1.49%) DV Cleared
7 17 (0.79%) Master's Degree
8 12 (0.56%) PMI Certification
9 10 (0.46%) PMP
10 9 (0.42%) AWS Certification
11 8 (0.37%) ISTQB
11 8 (0.37%) ISTQB Foundation Certification
12 6 (0.28%) Microsoft Certification
13 5 (0.23%) CCNA
13 5 (0.23%) Cisco Certification
13 5 (0.23%) CISSP
13 5 (0.23%) CompTIA Security+
13 5 (0.23%) MCSE
13 5 (0.23%) SANS
Quality Assurance & Compliance
1 93 (4.32%) Data Quality
2 58 (2.70%) QA
3 26 (1.21%) GDPR
4 5 (0.23%) ISO/IEC 27001
5 4 (0.19%) PCI DSS
6 3 (0.14%) MiFID
7 2 (0.093%) ISO 15022
7 2 (0.093%) ISO 20022
8 1 (0.046%) COBIT
System Software
1 391 (18.18%) Docker
2 179 (8.32%) Apache Flume
3 164 (7.62%) HDFS
4 107 (4.97%) Apache ZooKeeper
5 8 (0.37%) Active Directory
6 7 (0.33%) VirtualBox
7 4 (0.19%) VMware Infrastructure
8 2 (0.093%) Hyper-V
8 2 (0.093%) Virtual Machines
9 1 (0.046%) Ceph
9 1 (0.046%) Cronacle
9 1 (0.046%) Samba
9 1 (0.046%) Virtual Servers
Systems Management
1 244 (11.34%) Oozie
2 220 (10.23%) Kubernetes
3 205 (9.53%) Ansible
4 184 (8.55%) Hadoop YARN
5 153 (7.11%) Terraform
6 117 (5.44%) Opscode Chef
7 109 (5.07%) Puppet
8 56 (2.60%) Ambari
9 32 (1.49%) Mesos
10 31 (1.44%) Kibana
10 31 (1.44%) Prometheus
11 30 (1.39%) Nagios
12 26 (1.21%) Salt
13 18 (0.84%) Marathon
14 14 (0.65%) Docker Swarm
15 10 (0.46%) logstash
15 10 (0.46%) Microsoft Intune
15 10 (0.46%) SCCM
16 5 (0.23%) Control-M
17 4 (0.19%) Oracle ZFS
Vendors
1 355 (16.50%) Cloudera
2 248 (11.53%) Google
3 224 (10.41%) Tableau
4 220 (10.23%) Hortonworks
5 215 (10.00%) Oracle
6 167 (7.76%) Microsoft
7 154 (7.16%) Databricks
8 135 (6.28%) Informatica
9 91 (4.23%) MapR
10 88 (4.09%) Talend
11 73 (3.39%) IBM
12 48 (2.23%) SAP
13 41 (1.91%) Teradata
14 38 (1.77%) Splunk
15 30 (1.39%) Netezza
16 28 (1.30%) Vertica
17 27 (1.26%) CA Agile Central
18 25 (1.16%) SaltStack
18 25 (1.16%) TIBCO
19 24 (1.12%) Greenplum