Period
to 25 September 2018

The following table provides summary statistics for permanent job vacancies with a requirement for HDFS skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited HDFS over the 6 months to 25 September 2018 with a comparison to the same period in the previous 2 years.

Hadoop Distributed File System (HDFS)
UK
6 months to
25 Sep 2018
Same period 2017 Same period 2016
Rank 750 965 981
Rank change year-on-year +215 +16 +143
Permanent jobs citing HDFS 430 228 274
As % of all permanent IT jobs advertised in the UK 0.26% 0.13% 0.14%
As % of the System Software category 1.74% 1.00% 1.15%
Number of salaries quoted 320 180 214
UK median annual salary £70,000 £70,000 £80,000
Median salary % change year-on-year - -12.50% +14.29%
10th Percentile £41,100 £45,000 £45,000
90th Percentile £97,500 £102,500 £111,750
UK excluding London median annual salary £57,500 £57,500 £52,500
% change year-on-year - +9.52% -4.55%

HDFS is in the System Software category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for system software skills.

All System Software Skills
UK
Permanent vacancies with a requirement for system software skills 24,696 22,784 23,829
As % of all permanent IT jobs advertised in the UK 14.74% 12.92% 12.45%
Number of salaries quoted 20,199 19,178 20,172
UK median annual salary £49,500 £45,000 £42,500
Median salary % change year-on-year +10.00% +5.88% +6.25%
10th Percentile £24,750 £24,000 £23,250
90th Percentile £81,250 £75,000 £71,250
UK excluding London median annual salary £42,500 £38,750 £37,500
% change year-on-year +9.68% +3.33% -

HDFS
Job Vacancy Trend

Job postings citing HDFS as a percentage of all IT jobs advertised.

Job vacancy trend for HDFS in the UK

HDFS
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing HDFS.

Salary trend for HDFS in the UK

HDFS
Salary Histogram

The salary distribution of IT jobs citing HDFS over the 6 months to 25 September 2018.

Salary histogram for HDFS in the UK

HDFS
Top 14 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing HDFS within the UK over the 6 months to 25 September 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +185 406 £70,000 - 35
London +131 270 £75,000 +7.14% 21
UK excluding London +127 155 £57,500 - 14
South East +46 65 £57,500 +2.22% 9
East of England +51 30 £72,500 - 2
North of England +29 24 £40,000 -48.39%
Scotland +4 17 £52,500 -6.67%
Midlands +31 15 £60,000 +42.01% 1
West Midlands +39 13 £60,000 +42.01%
North West +8 13 £40,000 -48.39%
Yorkshire +37 11 £65,000 -
East Midlands +12 2 £67,500 +35.00% 1
South West +4 2 £57,500 +15.00% 2
Wales - 2 £58,000 -

For the 6 months to 25 September 2018, IT jobs citing HDFS also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for HDFS.

1 387 (90.00%) Hadoop
2 330 (76.74%) Apache Spark
3 292 (67.91%) Big Data
4 289 (67.21%) Apache Hive
5 235 (54.65%) Python
6 218 (50.70%) Java
7 198 (46.05%) SQL
8 193 (44.88%) HBase
9 185 (43.02%) Scala
10 166 (38.60%) Analytics
11 165 (38.37%) Kafka
12 161 (37.44%) Data Science
13 147 (34.19%) Amazon AWS
14 146 (33.95%) Linux
15 140 (32.56%) Machine Learning
16 133 (30.93%) ETL
17 122 (28.37%) Agile Software Development
18 115 (26.74%) Impala
19 100 (23.26%) Oozie
19 100 (23.26%) Sqoop
20 99 (23.02%) NoSQL
21 93 (21.63%) R
22 84 (19.53%) Apache
23 80 (18.60%) MapReduce
24 78 (18.14%) Hortonworks
25 77 (17.91%) Apache Flume
25 77 (17.91%) Apache Cassandra
25 77 (17.91%) Degree
26 68 (15.81%) Data Warehouse
26 68 (15.81%) Visualisation

HDFS
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 330 (76.74%) Apache Spark
2 84 (19.53%) Apache
3 53 (12.33%) Apache Pig
4 38 (8.84%) Elasticsearch
5 26 (6.05%) Apache Solr
5 26 (6.05%) SAS
6 8 (1.86%) Cloud Foundry
6 8 (1.86%) JBoss
6 8 (1.86%) WildFly
7 5 (1.16%) Tomcat
8 4 (0.93%) OpenStack
9 3 (0.70%) Oracle Workflow
10 2 (0.47%) Apache Geronimo
10 2 (0.47%) GemFire
10 2 (0.47%) GlassFish
10 2 (0.47%) WebSphere
11 1 (0.23%) IIS
Applications
1 6 (1.40%) MS Visio
2 3 (0.70%) Microsoft Project
3 2 (0.47%) CPLEX
3 2 (0.47%) SPSS
3 2 (0.47%) Weka
Cloud Services
1 147 (34.19%) Amazon AWS
2 55 (12.79%) Amazon EMR
2 55 (12.79%) Microsoft Azure
3 31 (7.21%) Amazon Kinesis
4 29 (6.74%) Serverless
5 26 (6.05%) AWS Lambda
6 22 (5.12%) Amazon S3
7 20 (4.65%) Azure Data Factory
8 19 (4.42%) Google Cloud Platform
9 13 (3.02%) Amazon EC2
10 11 (2.56%) GitHub
11 10 (2.33%) Cloud Computing
12 6 (1.40%) IaaS
13 5 (1.16%) PaaS
14 4 (0.93%) Loggly
14 4 (0.93%) Sumo Logic
15 2 (0.47%) Amazon ElastiCache
15 2 (0.47%) Amazon ELB
15 2 (0.47%) Virtual Private Cloud
16 1 (0.23%) Amazon CloudWatch
Communications & Networking
1 24 (5.58%) Sourcefire
2 17 (3.95%) Kerberos
3 12 (2.79%) Internet
4 11 (2.56%) Network Security
5 10 (2.33%) DNS
5 10 (2.33%) Firewall
5 10 (2.33%) SSL
6 9 (2.09%) TCP/IP
7 5 (1.16%) HTTP
8 3 (0.70%) VPN
9 2 (0.47%) VLAN
Database & Business Intelligence
1 387 (90.00%) Hadoop
2 292 (67.91%) Big Data
3 289 (67.21%) Apache Hive
4 193 (44.88%) HBase
5 115 (26.74%) Impala
6 100 (23.26%) Sqoop
7 99 (23.02%) NoSQL
8 77 (17.91%) Apache Cassandra
9 68 (15.81%) Data Lake
9 68 (15.81%) Data Warehouse
10 65 (15.12%) MongoDB
11 60 (13.95%) SQL Server
12 53 (12.33%) Amazon Redshift
13 47 (10.93%) Relational Database
14 33 (7.67%) SQL Server Integration Services
15 30 (6.98%) Parquet
16 29 (6.74%) MySQL
16 29 (6.74%) QlikView
17 28 (6.51%) Power BI
17 28 (6.51%) Text Mining
Development Applications
1 52 (12.09%) Jenkins
2 51 (11.86%) Git (software)
3 36 (8.37%) JUnit
4 10 (2.33%) Gradle
4 10 (2.33%) Maven
5 9 (2.09%) ERwin
5 9 (2.09%) IntelliJ
5 9 (2.09%) TeamCity
6 6 (1.40%) GitLab
7 4 (0.93%) Eclipse
7 4 (0.93%) Vagrant
8 3 (0.70%) Cucumber
8 3 (0.70%) git-flow
8 3 (0.70%) Grunt
8 3 (0.70%) JIRA
8 3 (0.70%) Selenium
9 2 (0.47%) Ant
9 2 (0.47%) FindBugs
9 2 (0.47%) Hudson
9 2 (0.47%) Subversion
General
1 55 (12.79%) Finance
2 42 (9.77%) Retail
3 35 (8.14%) Banking
4 33 (7.67%) Advertising
5 21 (4.88%) Marketing
5 21 (4.88%) Telecoms
6 14 (3.26%) Investment Banking
7 13 (3.02%) Pharmaceutical
8 11 (2.56%) Financial Institution
9 10 (2.33%) Front Office
10 8 (1.86%) Military
11 2 (0.47%) Automotive
12 1 (0.23%) Aerospace
12 1 (0.23%) Law
12 1 (0.23%) Legal
Job Titles
1 136 (31.63%) Data Engineer
2 104 (24.19%) Big Data Engineer
3 83 (19.30%) Developer
4 62 (14.42%) Data Scientist
5 38 (8.84%) Senior Data Scientist
6 37 (8.60%) Java Developer
7 32 (7.44%) Consultant
8 31 (7.21%) Hadoop Engineer
9 29 (6.74%) Architect
9 29 (6.74%) Hadoop Developer
10 28 (6.51%) Big Data Developer
11 25 (5.81%) Data Architect
12 24 (5.58%) Machine Learning Scientist
13 23 (5.35%) Senior Developer
14 21 (4.88%) Systems Engineer
15 20 (4.65%) Data Consultant
16 19 (4.42%) Java Engineer
16 19 (4.42%) Security Engineer
17 17 (3.95%) Software Engineer
18 16 (3.72%) Senior Java Developer
Libraries, Frameworks & Software Standards
1 165 (38.37%) Kafka
2 46 (10.70%) Elastic Stack
3 45 (10.47%) PySpark
4 38 (8.84%) Spring
5 35 (8.14%) TensorFlow
6 31 (7.21%) MLlib
7 27 (6.28%) Natural Language Processing
8 26 (6.05%) scikit-learn
8 26 (6.05%) Web Services
9 25 (5.81%) NumPy
9 25 (5.81%) Pandas
9 25 (5.81%) Theano
10 24 (5.58%) JSON
10 24 (5.58%) SciPy
11 22 (5.12%) Apache Avro
11 22 (5.12%) REST
12 21 (4.88%) Node.js
13 17 (3.95%) Middleware
14 16 (3.72%) .NET
15 14 (3.26%) RabbitMQ
Miscellaneous
1 65 (15.12%) Algorithms
2 59 (13.72%) Analytical Skills
3 51 (11.86%) Computer Science
4 49 (11.40%) Data Structures
5 37 (8.60%) Fintech
6 31 (7.21%) Greenfield Project
7 23 (5.35%) Virtual Team
8 16 (3.72%) Driving Licence
8 16 (3.72%) Life Science
8 16 (3.72%) Linux Command Line
9 11 (2.56%) Security Operations Centre
10 10 (2.33%) Cyberthreat
10 10 (2.33%) Equities
10 10 (2.33%) Management Information System
10 10 (2.33%) Replication
11 9 (2.09%) Derivative
11 9 (2.09%) Enterprise Software
12 8 (1.86%) PKI
13 4 (0.93%) Public Cloud
13 4 (0.93%) Taxonomies
Operating Systems
1 146 (33.95%) Linux
2 65 (15.12%) Unix
3 24 (5.58%) Windows
4 8 (1.86%) Ubuntu
5 7 (1.63%) Red Hat Enterprise Linux
6 6 (1.40%) CentOS
6 6 (1.40%) VMS
7 4 (0.93%) CoreOS
8 1 (0.23%) DC/OS
Processes & Methodologies
1 166 (38.60%) Analytics
2 161 (37.44%) Data Science
3 140 (32.56%) Machine Learning
4 133 (30.93%) ETL
5 122 (28.37%) Agile Software Development
6 80 (18.60%) MapReduce
7 68 (15.81%) Visualisation
8 66 (15.35%) Data Analysis
9 58 (13.49%) Problem-Solving
10 57 (13.26%) Continuous Integration
11 52 (12.09%) Data Engineering
11 52 (12.09%) Open Source
12 49 (11.40%) Software Engineering
13 47 (10.93%) Data Modelling
13 47 (10.93%) DevOps
14 46 (10.70%) Business Intelligence
14 46 (10.70%) Cybersecurity
14 46 (10.70%) TDD
15 39 (9.07%) Scrum
16 38 (8.84%) Data Visualisation
Programming Languages
1 235 (54.65%) Python
2 218 (50.70%) Java
3 198 (46.05%) SQL
4 185 (43.02%) Scala
5 93 (21.63%) R
6 40 (9.30%) Ruby
7 34 (7.91%) C
7 34 (7.91%) JavaScript
8 32 (7.44%) Bash Shell
9 26 (6.05%) HiveQL
10 22 (5.12%) Shell Script
11 21 (4.88%) C++
12 20 (4.65%) Perl
13 15 (3.49%) Go
14 13 (3.02%) PL/SQL
15 11 (2.56%) Java 8
16 9 (2.09%) C#
16 9 (2.09%) PHP
17 8 (1.86%) MATLAB
17 8 (1.86%) T-SQL
Qualifications
1 77 (17.91%) Degree
2 24 (5.58%) CEH
2 24 (5.58%) CISSP
2 24 (5.58%) GCIH
2 24 (5.58%) OSCP
3 23 (5.35%) Security Cleared
4 21 (4.88%) DV Cleared
5 18 (4.19%) Computer Science Degree
5 18 (4.19%) Numerate Degree
6 5 (1.16%) MCSA
6 5 (1.16%) Microsoft Certification
7 4 (0.93%) PhD
8 3 (0.70%) MCSE
9 2 (0.47%) MCSD
9 2 (0.47%) SC Cleared
10 1 (0.23%) AWS Certification
10 1 (0.23%) ITIL Foundation Certificate
Quality Assurance & Compliance
1 27 (6.28%) QA
2 14 (3.26%) Data Quality
3 11 (2.56%) GPG13
3 11 (2.56%) ISO/IEC 27001
4 1 (0.23%) GDPR
System Software
1 77 (17.91%) Apache Flume
2 35 (8.14%) Apache ZooKeeper
3 31 (7.21%) Docker
4 24 (5.58%) Snort
5 19 (4.42%) Virtual Machines
6 13 (3.02%) Active Directory
7 12 (2.79%) VMware Infrastructure
8 8 (1.86%) Hyper-V
9 4 (0.93%) VirtualBox
10 3 (0.70%) Oracle RAC
11 1 (0.23%) Xen
Systems Management
1 100 (23.26%) Oozie
2 51 (11.86%) Hadoop YARN
3 39 (9.07%) Kibana
4 31 (7.21%) Puppet
5 24 (5.58%) Suricata
6 23 (5.35%) Kubernetes
6 23 (5.35%) logstash
7 21 (4.88%) Grafana
7 21 (4.88%) Prometheus
8 17 (3.95%) Mesos
9 12 (2.79%) Ambari
10 11 (2.56%) Opscode Chef
10 11 (2.56%) QRadar
11 7 (1.63%) Ansible
12 5 (1.16%) Salt
12 5 (1.16%) ZABBIX
13 4 (0.93%) Docker Swarm
13 4 (0.93%) Nagios
13 4 (0.93%) Terraform
14 3 (0.70%) Oracle RMAN
Vendors
1 78 (18.14%) Hortonworks
2 60 (13.95%) Cloudera
3 56 (13.02%) Oracle
4 46 (10.70%) Tableau
5 36 (8.37%) Microsoft
6 27 (6.28%) WorldPay
7 24 (5.58%) TippingPoint
8 23 (5.35%) Talend
9 21 (4.88%) MapR
10 20 (4.65%) Google
11 19 (4.42%) Informatica
11 19 (4.42%) Sybase
12 17 (3.95%) SAP
13 16 (3.72%) Splunk
14 13 (3.02%) Atlassian
15 12 (2.79%) VMware
16 11 (2.56%) ArcSight
16 11 (2.56%) Fortinet
16 11 (2.56%) LogRhythm
16 11 (2.56%) Sophos