Period
to 14 November 2019

The following table provides summary statistics for permanent job vacancies with a requirement for HDFS skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited HDFS over the 6 months to 14 November 2019 with a comparison to the same period in the previous 2 years.

Hadoop Distributed File System (HDFS)
UK
6 months to
14 Nov 2019
Same period 2018 Same period 2017
Rank 844 804 964
Rank change year-on-year -40 +160 -16
Permanent jobs citing HDFS 211 347 253
As % of all permanent IT jobs advertised in the UK 0.15% 0.21% 0.14%
As % of the System Software category 0.90% 1.40% 1.08%
Number of salaries quoted 196 264 201
UK median annual salary £73,250 £70,000 £70,000
Median salary % change year-on-year +4.64% - -12.50%
10th Percentile £49,337 £45,000 £45,000
90th Percentile £91,875 £97,500 £107,500
UK excluding London median annual salary £57,275 £57,500 £57,500
% change year-on-year -0.39% - +4.55%

HDFS is in the System Software category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for system software skills.

All System Software Skills
UK
Permanent vacancies with a requirement for system software skills 23,575 24,780 23,365
As % of all permanent IT jobs advertised in the UK 17.19% 15.15% 12.93%
Number of salaries quoted 19,388 20,237 19,592
UK median annual salary £50,000 £50,000 £45,000
Median salary % change year-on-year - +11.11% +5.88%
10th Percentile £25,500 £25,000 £23,750
90th Percentile £85,000 £81,250 £75,000
UK excluding London median annual salary £42,500 £42,500 £39,250
% change year-on-year - +8.28% +4.67%

HDFS
Job Vacancy Trend

Job postings citing HDFS as a percentage of all IT jobs advertised.

Job vacancy trend for HDFS in the UK

HDFS
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing HDFS.

Salary trend for HDFS in the UK

HDFS
Salary Histogram

The salary distribution of IT jobs citing HDFS over the 6 months to 14 November 2019.

Salary histogram for HDFS in the UK

HDFS
Top 13 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing HDFS within the UK over the 6 months to 14 November 2019. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England -9 200 £75,000 +7.14% 18
London +11 140 £75,000 -1.64% 12
UK excluding London +16 71 £57,275 -0.39% 10
North of England +47 22 £57,275 -11.88% 2
Yorkshire +15 18 £57,275 -15.15% 1
East of England +45 17 £75,000 +3.45% 2
Scotland +21 11 £50,800 -5.49% 4
South West +12 10 £52,500 -1.41%
South East +12 7 £47,500 -13.64%
Midlands +44 4 £62,500 -16.67% 2
North East - 4 £57,275 - 1
West Midlands +29 2 £56,250 -6.25% 1
East Midlands +10 2 £67,500 -10.00% 1

For the 6 months to 14 November 2019, IT jobs citing HDFS also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for HDFS.

1 191 (90.52%) Hadoop
2 170 (80.57%) Apache Hive
3 162 (76.78%) Big Data
4 156 (73.93%) Apache Spark
5 146 (69.19%) Python
6 110 (52.13%) SQL
7 109 (51.66%) Azure
8 102 (48.34%) Agile Software Development
9 92 (43.60%) AWS
10 88 (41.71%) Scala
11 87 (41.23%) HBase
12 80 (37.91%) Linux
13 79 (37.44%) NoSQL
13 79 (37.44%) Impala
14 78 (36.97%) Data Modelling
15 72 (34.12%) Hadoop YARN
15 72 (34.12%) Data Engineering
16 70 (33.18%) Data Science
17 69 (32.70%) Business Intelligence
18 67 (31.75%) Java
19 61 (28.91%) Amazon S3
20 57 (27.01%) Apache
21 56 (26.54%) ETL
22 54 (25.59%) SQL Server
22 54 (25.59%) Amazon EMR
23 53 (25.12%) Analytics
24 50 (23.70%) Kafka
25 46 (21.80%) Sqoop
25 46 (21.80%) Data Warehouse
26 42 (19.91%) Oozie

HDFS
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 156 (73.93%) Apache Spark
2 57 (27.01%) Apache
3 21 (9.95%) Apache Airflow
4 14 (6.64%) Elasticsearch
5 12 (5.69%) Apache Pig
6 11 (5.21%) SAS
7 7 (3.32%) Apache Solr
8 6 (2.84%) Tomcat
8 6 (2.84%) WebLogic
9 1 (0.47%) Cloud Foundry
9 1 (0.47%) Confluence
9 1 (0.47%) JBoss
9 1 (0.47%) WebLogic Portal
Applications
1 6 (2.84%) Chatbot
2 1 (0.47%) Microsoft Project
Business Applications
1 1 (0.47%) Adobe Campaign
1 1 (0.47%) Dynamics CRM
Cloud Services
1 109 (51.66%) Azure
2 92 (43.60%) AWS
3 61 (28.91%) Amazon S3
4 54 (25.59%) Amazon EMR
5 28 (13.27%) GCP
6 20 (9.48%) Azure Data Factory
6 20 (9.48%) IaaS
6 20 (9.48%) PaaS
7 19 (9.00%) Amazon Kinesis
8 10 (4.74%) SaaS
9 8 (3.79%) Cloud Computing
9 8 (3.79%) GitHub
9 8 (3.79%) OpenShift
10 6 (2.84%) AWS CloudFormation
11 3 (1.42%) Amazon EC2
11 3 (1.42%) Power Platform
12 2 (0.95%) Serverless
13 1 (0.47%) AWS Lambda
13 1 (0.47%) Virtual Private Cloud
Communications & Networking
1 17 (8.06%) Firewall
2 16 (7.58%) Kerberos
3 12 (5.69%) VPN
4 9 (4.27%) SSL
5 7 (3.32%) HTTP
5 7 (3.32%) HTTPS
6 6 (2.84%) DNS
7 1 (0.47%) Broadband
Database & Business Intelligence
1 191 (90.52%) Hadoop
2 170 (80.57%) Apache Hive
3 162 (76.78%) Big Data
4 87 (41.23%) HBase
5 79 (37.44%) Impala
5 79 (37.44%) NoSQL
6 54 (25.59%) SQL Server
7 46 (21.80%) Data Warehouse
7 46 (21.80%) Sqoop
8 41 (19.43%) HDInsight
9 25 (11.85%) PostgreSQL
10 23 (10.90%) Data Lake
10 23 (10.90%) Neo4j
11 22 (10.43%) Data Mining
12 20 (9.48%) Apache Cassandra
12 20 (9.48%) Apache Kudu
12 20 (9.48%) Presto
13 19 (9.00%) Amazon Redshift
14 17 (8.06%) MongoDB
15 16 (7.58%) SQL Server Analysis Services
Development Applications
1 21 (9.95%) Git (software)
2 14 (6.64%) ProC
3 13 (6.16%) Jenkins
4 6 (2.84%) GitLab
5 5 (2.37%) Eclipse
5 5 (2.37%) IntelliJ
5 5 (2.37%) NetBeans
6 3 (1.42%) JIRA
7 2 (0.95%) Maven
8 1 (0.47%) Bitbucket
8 1 (0.47%) Gradle
8 1 (0.47%) Selenium
8 1 (0.47%) Sonatype Nexus
8 1 (0.47%) Team Foundation Server
8 1 (0.47%) Visual Studio Team System
General
1 20 (9.48%) Finance
2 16 (7.58%) Public Sector
3 12 (5.69%) Games
3 12 (5.69%) Legal
4 7 (3.32%) Retail
5 4 (1.90%) Manufacturing
5 4 (1.90%) Marketing
6 1 (0.47%) Banking
6 1 (0.47%) Front Office
6 1 (0.47%) Investment Banking
Job Titles
1 111 (52.61%) Data Engineer
2 44 (20.85%) Senior Data Engineer
3 40 (18.96%) Big Data Engineer
4 38 (18.01%) Architect
5 29 (13.74%) Data Architect
6 26 (12.32%) Cloud Architect
7 21 (9.95%) Data Scientist
8 20 (9.48%) Developer
9 17 (8.06%) Senior Data Scientist
10 12 (5.69%) DevOps Engineer
10 12 (5.69%) Hadoop Engineer
11 11 (5.21%) Cloud Engineer
12 10 (4.74%) Java Developer
12 10 (4.74%) Senior Hadoop Engineer
13 8 (3.79%) Big Data Developer
13 8 (3.79%) Senior DevOps
13 8 (3.79%) Senior DevOps Engineer
14 7 (3.32%) Hadoop Architect
15 6 (2.84%) Senior Developer
15 6 (2.84%) Senior Java Developer
Libraries, Frameworks & Software Standards
1 50 (23.70%) Kafka
2 24 (11.37%) LAPP Stack
3 16 (7.58%) D3.js
3 16 (7.58%) ggplot2
4 15 (7.11%) Lambda Expressions
5 14 (6.64%) Elastic Stack
6 12 (5.69%) OAuth
6 12 (5.69%) SAML
7 11 (5.21%) LDAP
8 10 (4.74%) Web Services
9 9 (4.27%) Spring
10 7 (3.32%) Middleware
10 7 (3.32%) RabbitMQ
11 5 (2.37%) ActiveMQ
11 5 (2.37%) Hibernate
11 5 (2.37%) JPA
11 5 (2.37%) Node.js
11 5 (2.37%) SOAP
11 5 (2.37%) Spring Boot
11 5 (2.37%) Spring Cloud
Miscellaneous
1 28 (13.27%) Algorithms
2 26 (12.32%) Analytical Skills
3 12 (5.69%) Data Centre
4 10 (4.74%) Management Information System
5 8 (3.79%) Cloud Native
5 8 (3.79%) Public Cloud
5 8 (3.79%) Self-Motivation
6 5 (2.37%) Greenfield Project
7 4 (1.90%) Data Structures
8 3 (1.42%) Fixed Income
8 3 (1.42%) Replication
9 2 (0.95%) Equities
9 2 (0.95%) Hybrid Cloud
9 2 (0.95%) Linux Command Line
9 2 (0.95%) Tandem
10 1 (0.47%) Embedded Systems
10 1 (0.47%) Market Risk
10 1 (0.47%) Social Network
10 1 (0.47%) Virtual Team
Operating Systems
1 80 (37.91%) Linux
2 30 (14.22%) Unix
3 2 (0.95%) SUSE Linux Enterprise Server
3 2 (0.95%) Windows
Processes & Methodologies
1 102 (48.34%) Agile Software Development
2 78 (36.97%) Data Modelling
3 72 (34.12%) Data Engineering
4 70 (33.18%) Data Science
5 69 (32.70%) Business Intelligence
6 56 (26.54%) ETL
7 53 (25.12%) Analytics
8 41 (19.43%) Machine Learning
9 30 (14.22%) Software Engineering
10 28 (13.27%) Data Integration
11 25 (11.85%) Data Management
11 25 (11.85%) MapReduce
12 24 (11.37%) DevOps
13 22 (10.43%) Data Development
13 22 (10.43%) Metadata Management
13 22 (10.43%) Microservices
13 22 (10.43%) Problem-Solving
13 22 (10.43%) Visualisation
14 20 (9.48%) Coaching
14 20 (9.48%) Test Automation
Programming Languages
1 146 (69.19%) Python
2 110 (52.13%) SQL
3 88 (41.71%) Scala
4 67 (31.75%) Java
5 36 (17.06%) Shell Script
6 22 (10.43%) R
7 19 (9.00%) JavaScript
7 19 (9.00%) PL/SQL
8 18 (8.53%) Bash Shell
9 17 (8.06%) Ruby
10 14 (6.64%) C
11 6 (2.84%) Perl
11 6 (2.84%) T-SQL
12 4 (1.90%) Groovy
12 4 (1.90%) Java 8
13 2 (0.95%) PHP
14 1 (0.47%) HiveQL
14 1 (0.47%) PHP 7
14 1 (0.47%) Pig Latin
Qualifications
1 18 (8.53%) Degree
2 5 (2.37%) Computer Science Degree
3 2 (0.95%) MCSA
3 2 (0.95%) Microsoft Certification
3 2 (0.95%) Security Cleared
4 1 (0.47%) SC Cleared
Quality Assurance & Compliance
1 39 (18.48%) Data Quality
2 31 (14.69%) QA
3 13 (6.16%) GDPR
4 12 (5.69%) ISO/IEC 27001
System Software
1 22 (10.43%) Docker
2 21 (9.95%) Apache Flume
3 13 (6.16%) Apache ZooKeeper
4 2 (0.95%) Cronacle
Systems Management
1 72 (34.12%) Hadoop YARN
2 42 (19.91%) Oozie
3 15 (7.11%) Ambari
4 12 (5.69%) Single Sign-On
5 11 (5.21%) Kubernetes
5 11 (5.21%) Terraform
6 8 (3.79%) Ansible
7 5 (2.37%) Puppet
8 4 (1.90%) Mesos
9 2 (0.95%) Control-M
10 1 (0.47%) Grafana
10 1 (0.47%) Prometheus
Vendors
1 40 (18.96%) Cloudera
2 28 (13.27%) Oracle
3 19 (9.00%) Hortonworks
4 18 (8.53%) Google
5 17 (8.06%) Teradata
6 15 (7.11%) Databricks
7 8 (3.79%) Microsoft
8 7 (3.32%) Informatica
9 5 (2.37%) Tableau
10 3 (1.42%) MapR
10 3 (1.42%) Talend
11 2 (0.95%) Blue Prism
11 2 (0.95%) Salesforce.com
11 2 (0.95%) UiPath
12 1 (0.47%) Adobe
12 1 (0.47%) Alteryx
12 1 (0.47%) BEA
12 1 (0.47%) Confluent
12 1 (0.47%) IBM
12 1 (0.47%) Salesforce Marketing Cloud