Period
to 15 December 2018

The following table provides summary statistics for permanent job vacancies with a requirement for HDFS skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited HDFS over the 6 months to 15 December 2018 with a comparison to the same period in the previous 2 years.

Hadoop Distributed File System (HDFS)
UK
6 months to
15 Dec 2018
Same period 2017 Same period 2016
Rank 818 968 946
Rank change year-on-year +150 -22 +121
Permanent jobs citing HDFS 320 258 287
As % of all permanent IT jobs advertised in the UK 0.20% 0.14% 0.15%
As % of the System Software category 1.32% 1.11% 1.26%
Number of salaries quoted 246 217 215
UK median annual salary £72,500 £70,000 £77,500
Median salary % change year-on-year +3.57% -9.68% +10.71%
10th Percentile £51,375 £45,000 £49,750
90th Percentile £97,500 £102,500 £104,500
UK excluding London median annual salary £60,000 £57,500 £60,000
% change year-on-year +4.35% -4.17% -

HDFS is in the System Software category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for system software skills.

All System Software Skills
UK
Permanent vacancies with a requirement for system software skills 24,229 23,269 22,858
As % of all permanent IT jobs advertised in the UK 15.33% 12.97% 12.34%
Number of salaries quoted 19,738 19,383 19,123
UK median annual salary £50,000 £45,000 £42,500
Median salary % change year-on-year +11.11% +5.88% +6.25%
10th Percentile £25,000 £24,000 £23,500
90th Percentile £82,500 £75,000 £72,500
UK excluding London median annual salary £42,500 £40,000 £37,500
% change year-on-year +6.25% +6.67% -

HDFS
Job Vacancy Trend

Job postings citing HDFS as a percentage of all IT jobs advertised.

Job vacancy trend for HDFS in the UK

HDFS
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing HDFS.

Salary trend for HDFS in the UK

HDFS
Salary Histogram

The salary distribution of IT jobs citing HDFS over the 6 months to 15 December 2018.

Salary histogram for HDFS in the UK

HDFS
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing HDFS within the UK over the 6 months to 15 December 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +117 308 £72,500 +3.57% 27
London +88 212 £75,000 - 17
UK excluding London +78 108 £60,000 +4.35% 10
East of England +54 26 £72,500 -21.62% 1
South East -7 26 £50,000 -13.04% 2
North of England +32 19 £62,500 +38.89% 6
South West +13 14 £53,126 +6.25% 1
Midlands +51 11 £75,000 +62.60%
Scotland +15 9 £57,500 -
North West +21 8 £70,000 +55.56% 2
Yorkshire - 8 £65,000 - 4
East Midlands +26 6 £75,000 +50.00%
West Midlands +51 5 £60,000 +42.01%
Wales - 3 £54,000 -
North East - 3 £49,337 -

For the 6 months to 15 December 2018, IT jobs citing HDFS also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for HDFS.

1 272 (85.00%) Hadoop
2 252 (78.75%) Apache Spark
3 229 (71.56%) Big Data
4 223 (69.69%) Apache Hive
5 188 (58.75%) Java
6 169 (52.81%) Python
7 155 (48.44%) HBase
7 155 (48.44%) Kafka
8 146 (45.63%) Agile Software Development
9 139 (43.44%) Scala
10 123 (38.44%) SQL
11 120 (37.50%) Linux
11 120 (37.50%) Data Science
12 109 (34.06%) Amazon AWS
13 108 (33.75%) NoSQL
14 97 (30.31%) ETL
15 96 (30.00%) Machine Learning
16 94 (29.38%) Apache Cassandra
17 91 (28.44%) Analytics
18 80 (25.00%) MongoDB
18 80 (25.00%) Impala
19 76 (23.75%) Oozie
20 75 (23.44%) Apache
21 73 (22.81%) Unix
22 72 (22.50%) Hortonworks
23 71 (22.19%) Degree
24 67 (20.94%) Open Source
25 61 (19.06%) Cloudera
26 60 (18.75%) Git (software)
26 60 (18.75%) Analytical Skills

HDFS
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 252 (78.75%) Apache Spark
2 75 (23.44%) Apache
3 48 (15.00%) Apache Pig
4 44 (13.75%) Elasticsearch
5 33 (10.31%) Apache Solr
6 25 (7.81%) SAS
7 16 (5.00%) JBoss
8 12 (3.75%) Tomcat
9 11 (3.44%) WildFly
10 10 (3.13%) Apache Geronimo
10 10 (3.13%) GlassFish
10 10 (3.13%) WebSphere
11 8 (2.50%) Cloud Foundry
12 6 (1.88%) Jetty
13 2 (0.63%) GemFire
13 2 (0.63%) OpenStack
14 1 (0.31%) Apache Airflow
Applications
1 10 (3.13%) Weka
2 1 (0.31%) CPLEX
2 1 (0.31%) Microsoft Office
2 1 (0.31%) Microsoft Project
2 1 (0.31%) SPSS
Cloud Services
1 109 (34.06%) Amazon AWS
2 40 (12.50%) Amazon EMR
3 37 (11.56%) Microsoft Azure
4 33 (10.31%) GitHub
5 27 (8.44%) Serverless
6 26 (8.13%) AWS Lambda
7 23 (7.19%) Cloud Computing
8 20 (6.25%) Amazon EC2
9 19 (5.94%) Amazon Kinesis
10 15 (4.69%) Amazon S3
11 12 (3.75%) Google Cloud Platform
12 9 (2.81%) Azure Data Factory
13 6 (1.88%) SaaS
14 5 (1.56%) AWS CloudFormation
15 3 (0.94%) Azure Machine Learning
16 2 (0.63%) Amazon ElastiCache
16 2 (0.63%) Amazon ELB
16 2 (0.63%) AWS Data Pipeline
16 2 (0.63%) Sumo Logic
16 2 (0.63%) Virtual Private Cloud
Communications & Networking
1 12 (3.75%) Kerberos
2 9 (2.81%) Firewall
2 9 (2.81%) SSL
3 8 (2.50%) Internet
3 8 (2.50%) Sourcefire
3 8 (2.50%) TCP/IP
4 7 (2.19%) DNS
5 6 (1.88%) HTTP
6 4 (1.25%) Network Security
7 3 (0.94%) VPN
8 2 (0.63%) VLAN
9 1 (0.31%) WAN
Database & Business Intelligence
1 272 (85.00%) Hadoop
2 229 (71.56%) Big Data
3 223 (69.69%) Apache Hive
4 155 (48.44%) HBase
5 108 (33.75%) NoSQL
6 94 (29.38%) Apache Cassandra
7 80 (25.00%) Impala
7 80 (25.00%) MongoDB
8 54 (16.88%) Sqoop
9 52 (16.25%) Data Warehouse
9 52 (16.25%) Relational Database
10 46 (14.38%) SQL Server
11 45 (14.06%) MySQL
12 44 (13.75%) PostgreSQL
13 37 (11.56%) Data Lake
14 33 (10.31%) Amazon Redshift
15 30 (9.38%) Spark SQL
16 28 (8.75%) RDBMS
17 23 (7.19%) Redis
18 22 (6.88%) Flink
Development Applications
1 60 (18.75%) Git (software)
2 57 (17.81%) Jenkins
3 30 (9.38%) Maven
4 26 (8.13%) JUnit
5 20 (6.25%) IntelliJ
6 18 (5.63%) Gradle
7 15 (4.69%) Eclipse
8 10 (3.13%) Ant
9 9 (2.81%) ERwin
10 7 (2.19%) JIRA
11 6 (1.88%) Subversion
11 6 (1.88%) Team Foundation Server
12 5 (1.56%) NetBeans
13 4 (1.25%) FindBugs
13 4 (1.25%) TeamCity
14 3 (0.94%) Selenium
15 2 (0.63%) Cucumber
15 2 (0.63%) Grunt
15 2 (0.63%) Hudson
15 2 (0.63%) Vagrant
General
1 58 (18.13%) Finance
2 37 (11.56%) Banking
3 25 (7.81%) Telecoms
4 23 (7.19%) Retail
5 17 (5.31%) Advertising
6 14 (4.38%) Pharmaceutical
7 7 (2.19%) Marketing
8 5 (1.56%) Investment Banking
9 4 (1.25%) Financial Institution
9 4 (1.25%) Law
10 3 (0.94%) Electronics
11 2 (0.63%) Front Office
11 2 (0.63%) Manufacturing
12 1 (0.31%) Legal
12 1 (0.31%) Military
Job Titles
1 100 (31.25%) Data Engineer
1 100 (31.25%) Developer
2 73 (22.81%) Big Data Engineer
3 56 (17.50%) Java Developer
4 52 (16.25%) Big Data Developer
5 32 (10.00%) Data Scientist
6 30 (9.38%) Senior Developer
7 27 (8.44%) Senior Data Scientist
8 26 (8.13%) Hadoop Developer
9 20 (6.25%) Architect
10 19 (5.94%) Senior Java Developer
11 15 (4.69%) Apache Spark Developer
11 15 (4.69%) Machine Learning Scientist
12 14 (4.38%) Core Java Developer
13 12 (3.75%) Lead Developer
13 12 (3.75%) Lead Java Developer
14 11 (3.44%) Hadoop Engineer
14 11 (3.44%) Java Engineer
14 11 (3.44%) Senior Big Data Developer
14 11 (3.44%) Systems Engineer
Libraries, Frameworks & Software Standards
1 155 (48.44%) Kafka
2 45 (14.06%) Spring
3 39 (12.19%) PySpark
4 37 (11.56%) Web Services
5 31 (9.69%) REST
6 27 (8.44%) Elastic Stack
7 25 (7.81%) Apache Avro
7 25 (7.81%) TensorFlow
8 22 (6.88%) Hibernate
9 21 (6.56%) Natural Language Processing
9 21 (6.56%) React
10 19 (5.94%) MLlib
10 19 (5.94%) RabbitMQ
10 19 (5.94%) scikit-learn
10 19 (5.94%) Spark Streaming
10 19 (5.94%) Spring Boot
11 17 (5.31%) HTML
11 17 (5.31%) NumPy
12 16 (5.00%) CSS
12 16 (5.00%) SciPy
Miscellaneous
1 60 (18.75%) Analytical Skills
2 39 (12.19%) Algorithms
3 24 (7.50%) Data Structures
4 23 (7.19%) Virtual Team
5 20 (6.25%) Greenfield Project
6 16 (5.00%) Life Science
7 13 (4.06%) Fintech
8 12 (3.75%) Linux Command Line
9 9 (2.81%) Enterprise Software
10 7 (2.19%) Cloud Native
11 6 (1.88%) Driving Licence
12 4 (1.25%) Data Centre
12 4 (1.25%) Mainframe
12 4 (1.25%) Replication
12 4 (1.25%) User Experience
13 3 (0.94%) Derivative
13 3 (0.94%) Enterprise Cloud
13 3 (0.94%) Security Operations Centre
14 2 (0.63%) Cyberthreat
14 2 (0.63%) Self-Motivation
Operating Systems
1 120 (37.50%) Linux
2 73 (22.81%) Unix
3 18 (5.63%) Windows
4 11 (3.44%) Ubuntu
5 7 (2.19%) Red Hat Enterprise Linux
6 5 (1.56%) VMS
7 4 (1.25%) CentOS
8 2 (0.63%) CoreOS
9 1 (0.31%) DC/OS
Processes & Methodologies
1 146 (45.63%) Agile Software Development
2 120 (37.50%) Data Science
3 97 (30.31%) ETL
4 96 (30.00%) Machine Learning
5 91 (28.44%) Analytics
6 67 (20.94%) Open Source
7 57 (17.81%) Computer Science
8 56 (17.50%) Continuous Integration
8 56 (17.50%) MapReduce
9 49 (15.31%) TDD
10 48 (15.00%) Software Engineering
11 45 (14.06%) Data Analysis
12 43 (13.44%) Data Engineering
13 40 (12.50%) Data Modelling
14 39 (12.19%) DevOps
14 39 (12.19%) Scrum
15 38 (11.88%) Problem-Solving
16 34 (10.63%) Visualisation
17 32 (10.00%) Data Ingestion
18 31 (9.69%) Data Integration
Programming Languages
1 188 (58.75%) Java
2 169 (52.81%) Python
3 139 (43.44%) Scala
4 123 (38.44%) SQL
5 53 (16.56%) JavaScript
6 48 (15.00%) R
7 42 (13.13%) Ruby
8 29 (9.06%) Shell Script
9 20 (6.25%) Java 8
10 16 (5.00%) Bash Shell
11 15 (4.69%) C++
11 15 (4.69%) T-SQL
12 12 (3.75%) C#
13 11 (3.44%) C
13 11 (3.44%) Groovy
13 11 (3.44%) PL/SQL
14 9 (2.81%) PHP
15 7 (2.19%) Go
16 4 (1.25%) HiveQL
16 4 (1.25%) Perl
Qualifications
1 71 (22.19%) Degree
2 26 (8.13%) Computer Science Degree
3 10 (3.13%) Security Cleared
4 8 (2.50%) CEH
4 8 (2.50%) CISSP
4 8 (2.50%) DV Cleared
4 8 (2.50%) GCIH
4 8 (2.50%) OSCP
5 3 (0.94%) MCSA
5 3 (0.94%) MCSE
5 3 (0.94%) Microsoft Certification
5 3 (0.94%) Numerate Degree
6 2 (0.63%) PhD
6 2 (0.63%) SC Cleared
7 1 (0.31%) AWS Certification
Quality Assurance & Compliance
1 20 (6.25%) QA
2 6 (1.88%) Data Quality
2 6 (1.88%) IFRS
3 5 (1.56%) GDPR
4 3 (0.94%) GPG13
4 3 (0.94%) ISO/IEC 27001
5 2 (0.63%) PCI DSS
6 1 (0.31%) EMIR
6 1 (0.31%) MiFID
System Software
1 50 (15.63%) Apache Flume
2 39 (12.19%) Apache ZooKeeper
3 35 (10.94%) Docker
4 15 (4.69%) Virtual Machines
5 9 (2.81%) Active Directory
6 8 (2.50%) Snort
7 3 (0.94%) VMware Infrastructure
8 2 (0.63%) VirtualBox
9 1 (0.31%) Hyper-V
Systems Management
1 76 (23.75%) Oozie
2 40 (12.50%) Kibana
3 38 (11.88%) Hadoop YARN
4 31 (9.69%) Puppet
5 28 (8.75%) Kubernetes
6 25 (7.81%) logstash
7 19 (5.94%) Grafana
7 19 (5.94%) Prometheus
8 16 (5.00%) Mesos
9 8 (2.50%) Suricata
10 7 (2.19%) Ansible
10 7 (2.19%) Opscode Chef
11 5 (1.56%) Ambari
12 4 (1.25%) Rundeck
13 3 (0.94%) Graphite
13 3 (0.94%) QRadar
13 3 (0.94%) Terraform
14 2 (0.63%) Nagios
14 2 (0.63%) Salt
14 2 (0.63%) ZABBIX
Vendors
1 72 (22.50%) Hortonworks
2 61 (19.06%) Cloudera
3 57 (17.81%) Oracle
4 38 (11.88%) Tableau
5 28 (8.75%) MapR
6 18 (5.63%) Microsoft
7 15 (4.69%) Google
8 12 (3.75%) Cognizant
8 12 (3.75%) SAP
9 11 (3.44%) Talend
10 10 (3.13%) Alteryx
10 10 (3.13%) Salesforce.com
11 8 (2.50%) TippingPoint
12 7 (2.19%) Splunk
13 6 (1.88%) Atlassian
14 5 (1.56%) CA
14 5 (1.56%) Databricks
14 5 (1.56%) IBM
14 5 (1.56%) Teradata
15 4 (1.25%) Informatica