Period
to 30 March 2020

The following table provides summary statistics for permanent job vacancies with a requirement for HDFS skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited HDFS over the 6 months to 30 March 2020 with a comparison to the same period in the previous 2 years.

Hadoop Distributed File System (HDFS)
UK
6 months to
30 Mar 2020
Same period 2019 Same period 2018
Rank 863 868 843
Rank change year-on-year +5 -25 +85
Permanent jobs citing HDFS 130 257 387
As % of all permanent jobs advertised in the UK 0.11% 0.17% 0.22%
As % of the System Software category 0.64% 1.08% 1.60%
Number of salaries quoted 124 211 303
Median annual salary £75,000 £72,500 £70,000
Median salary % change year-on-year +3.45% +3.57% -6.67%
10th Percentile £50,000 £49,337 £40,500
90th Percentile £85,000 £100,000 £97,500
UK excluding London median annual salary £72,500 £59,000 £60,000
% change year-on-year +22.88% -1.67% -11.11%

HDFS is in the System Software category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for system software skills.

All System Software Skills
UK
Permanent vacancies with a requirement for system software skills 20,196 23,791 24,164
As % of all permanent IT jobs advertised in the UK 17.06% 15.95% 13.63%
Number of salaries quoted 16,398 19,342 19,773
Median annual salary £50,000 £50,000 £45,000
Median salary % change year-on-year - +11.11% +5.88%
10th Percentile £26,250 £25,750 £24,750
90th Percentile £85,000 £82,500 £77,500
UK excluding London median annual salary £44,000 £42,500 £40,000
% change year-on-year +3.53% +6.25% +6.67%

HDFS
Job Vacancy Trend

Job postings citing HDFS as a proportion of all IT jobs advertised.

Job vacancy trend for HDFS in the UK

HDFS
Salary Trend

3-month moving average salary quoted in jobs citing HDFS.

Salary trend for HDFS in the UK

HDFS
Salary Histogram

Salary distribution for jobs citing HDFS over the 6 months to 30 March 2020.

Salary histogram for HDFS in the UK

HDFS
Top 15 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing HDFS within the UK over the 6 months to 30 March 2020. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +1 122 £75,000 +3.45% 15
UK excluding London +79 75 £72,500 +22.88% 9
London -19 55 £77,500 +3.33% 6
Midlands +73 25 £75,000 -1.64% 6
East Midlands +31 20 £75,000 -1.64% 1
North of England +56 17 £54,671 +10.81% 1
East of England +51 16 £75,000 +3.45%
Yorkshire +40 11 £54,835 +11.14% 1
Work from Home +7 10 £75,000 -25.00% 1
Scotland +48 8 £54,000 -15.29%
South East +66 7 £75,000 +76.47% 1
West Midlands - 5 £52,500 - 5
North East +4 4 £54,671 +10.81%
North West +45 2 - -
South West +30 2 £77,500 +55.00% 1

For the 6 months to 30 March 2020, IT jobs citing HDFS also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for HDFS.

1 127 (97.69%) Hadoop
2 121 (93.08%) Big Data
3 106 (81.54%) Apache Hive
4 95 (73.08%) Apache Spark
5 73 (56.15%) Agile Software Development
6 66 (50.77%) Python
7 62 (47.69%) HBase
8 61 (46.92%) Scala
8 61 (46.92%) Java
9 59 (45.38%) Impala
10 54 (41.54%) AWS
11 51 (39.23%) SQL
12 50 (38.46%) Azure
13 49 (37.69%) Kafka
13 49 (37.69%) Analytics
13 49 (37.69%) Data Engineering
14 45 (34.62%) Data Modelling
15 44 (33.85%) NoSQL
16 39 (30.00%) Hadoop YARN
17 38 (29.23%) Software Engineering
18 34 (26.15%) Linux
18 34 (26.15%) Data Science
19 32 (24.62%) Amazon EMR
19 32 (24.62%) Cloudera
20 31 (23.85%) Business Intelligence
21 30 (23.08%) MapReduce
22 29 (22.31%) ETL
23 28 (21.54%) SQL Server
23 28 (21.54%) Amazon S3
24 25 (19.23%) Machine Learning

HDFS
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 95 (73.08%) Apache Spark
2 22 (16.92%) Apache
3 8 (6.15%) Elasticsearch
4 7 (5.38%) Apache Airflow
5 5 (3.85%) Apache Pig
5 5 (3.85%) Tomcat
5 5 (3.85%) WebLogic
6 4 (3.08%) SAS
7 2 (1.54%) CMS
8 1 (0.77%) Jupyter
8 1 (0.77%) OpenStack
8 1 (0.77%) OpenWhisk
Applications
1 4 (3.08%) Chatbot
2 1 (0.77%) Microsoft Excel
2 1 (0.77%) SPSS
Business Applications
1 1 (0.77%) Oracle Applications
Cloud Services
1 54 (41.54%) AWS
2 50 (38.46%) Azure
3 32 (24.62%) Amazon EMR
4 28 (21.54%) Amazon S3
5 15 (11.54%) Amazon Kinesis
6 10 (7.69%) IaaS
6 10 (7.69%) PaaS
7 9 (6.92%) Azure Data Factory
7 9 (6.92%) GCP
8 6 (4.62%) Cloud Computing
9 4 (3.08%) OpenShift
9 4 (3.08%) SaaS
9 4 (3.08%) Serverless
10 3 (2.31%) Adobe Creative Cloud
11 2 (1.54%) Power Platform
Communications & Networking
1 9 (6.92%) Firewall
2 7 (5.38%) SSL
3 6 (4.62%) Kerberos
3 6 (4.62%) VPN
4 4 (3.08%) DNS
4 4 (3.08%) HTTP
4 4 (3.08%) HTTPS
5 2 (1.54%) FTP
6 1 (0.77%) IPFS
Database & Business Intelligence
1 127 (97.69%) Hadoop
2 121 (93.08%) Big Data
3 106 (81.54%) Apache Hive
4 62 (47.69%) HBase
5 59 (45.38%) Impala
6 44 (33.85%) NoSQL
7 28 (21.54%) SQL Server
8 25 (19.23%) Data Warehouse
9 21 (16.15%) Sqoop
10 19 (14.62%) HDInsight
11 16 (12.31%) Data Lake
12 15 (11.54%) Apache Kudu
13 14 (10.77%) Amazon Redshift
13 14 (10.77%) Relational Database
14 13 (10.00%) RDBMS
15 11 (8.46%) Neo4j
16 10 (7.69%) OLTP
16 10 (7.69%) PostgreSQL
17 9 (6.92%) Amazon DynamoDB
18 8 (6.15%) Apache Cassandra
Development Applications
1 9 (6.92%) Git (software)
2 8 (6.15%) Jenkins
3 3 (2.31%) JIRA
3 3 (2.31%) JUnit
3 3 (2.31%) Octopus Deploy
3 3 (2.31%) TeamCity
3 3 (2.31%) Visual Studio Team System
4 1 (0.77%) Artifactory
4 1 (0.77%) GitLab
4 1 (0.77%) IBM UrbanCode
4 1 (0.77%) Maven
4 1 (0.77%) ProC
4 1 (0.77%) Sonatype Nexus
4 1 (0.77%) Team Foundation Server
General
1 23 (17.69%) Finance
2 13 (10.00%) Retail
3 11 (8.46%) Public Sector
4 7 (5.38%) Legal
5 5 (3.85%) Banking
6 4 (3.08%) Marketing
7 2 (1.54%) Manufacturing
8 1 (0.77%) Games
8 1 (0.77%) Telecoms
Job Titles
1 60 (46.15%) Data Engineer
2 31 (23.85%) Architect
3 29 (22.31%) Big Data Engineer
4 23 (17.69%) Data Architect
5 18 (13.85%) Senior Data Engineer
6 16 (12.31%) Developer
7 15 (11.54%) Cloud Architect
7 15 (11.54%) Java Developer
8 12 (9.23%) Analytics Developer
8 12 (9.23%) Big Data Developer
8 12 (9.23%) Lead Developer
8 12 (9.23%) Lead Java Developer
8 12 (9.23%) Senior Big Data Developer
8 12 (9.23%) Senior Developer
8 12 (9.23%) Senior Java Developer
9 10 (7.69%) Big Data Architect
10 8 (6.15%) DevOps Engineer
11 7 (5.38%) Hadoop Architect
11 7 (5.38%) Technical Architect
12 6 (4.62%) Senior Data Scientist
Libraries, Frameworks & Software Standards
1 49 (37.69%) Kafka
2 15 (11.54%) J2EE
3 10 (7.69%) Elastic Stack
3 10 (7.69%) LAPP Stack
4 9 (6.92%) Lambda Expressions
5 7 (5.38%) OAuth
6 6 (4.62%) LDAP
6 6 (4.62%) PySpark
6 6 (4.62%) SAML
7 5 (3.85%) .NET
8 4 (3.08%) HTML
8 4 (3.08%) Middleware
9 3 (2.31%) Web Services
10 2 (1.54%) Akka
10 2 (1.54%) D3.js
10 2 (1.54%) ggplot2
11 1 (0.77%) Flask
11 1 (0.77%) GraphQL
11 1 (0.77%) Pandas
11 1 (0.77%) Spark Streaming
Miscellaneous
1 18 (13.85%) Analytical Skills
2 17 (13.08%) Public Cloud
3 16 (12.31%) Algorithms
4 13 (10.00%) Enterprise Software
5 6 (4.62%) Data Centre
6 5 (3.85%) Cloud Native
7 4 (3.08%) Self-Motivation
8 3 (2.31%) Derivative
8 3 (2.31%) Greenfield Project
8 3 (2.31%) Management Information System
8 3 (2.31%) Mobile App
9 2 (1.54%) Tandem
10 1 (0.77%) Embedded Systems
10 1 (0.77%) PMI
Operating Systems
1 34 (26.15%) Linux
2 18 (13.85%) Unix
3 3 (2.31%) Android
3 3 (2.31%) Apple iOS
4 2 (1.54%) Red Hat Enterprise Linux
Processes & Methodologies
1 73 (56.15%) Agile Software Development
2 49 (37.69%) Analytics
2 49 (37.69%) Data Engineering
3 45 (34.62%) Data Modelling
4 38 (29.23%) Software Engineering
5 34 (26.15%) Data Science
6 31 (23.85%) Business Intelligence
7 30 (23.08%) MapReduce
8 29 (22.31%) ETL
9 25 (19.23%) Machine Learning
10 19 (14.62%) Data Analysis
10 19 (14.62%) Data Development
10 19 (14.62%) Scrum
11 18 (13.85%) Technical Leadership
12 17 (13.08%) DevOps
12 17 (13.08%) Performance Tuning
13 16 (12.31%) Test Automation
14 15 (11.54%) Mentoring
15 13 (10.00%) Coaching
15 13 (10.00%) Microservices
Programming Languages
1 66 (50.77%) Python
2 61 (46.92%) Java
2 61 (46.92%) Scala
3 51 (39.23%) SQL
4 21 (16.15%) Shell Script
5 10 (7.69%) Bash Shell
6 9 (6.92%) Ruby
7 7 (5.38%) R
8 5 (3.85%) JavaScript
9 4 (3.08%) HiveQL
9 4 (3.08%) PL/SQL
10 3 (2.31%) Java 8
11 2 (1.54%) C
11 2 (1.54%) Groovy
11 2 (1.54%) Haskell
11 2 (1.54%) Perl
12 1 (0.77%) Go
12 1 (0.77%) Kotlin
12 1 (0.77%) PHP
Qualifications
1 20 (15.38%) Degree
2 12 (9.23%) AWS Certification
3 3 (2.31%) Computer Science Degree
4 2 (1.54%) Master's Degree
5 1 (0.77%) PMI Certification
5 1 (0.77%) SC Cleared
5 1 (0.77%) Security Cleared
Quality Assurance & Compliance
1 19 (14.62%) Data Quality
2 13 (10.00%) QA
3 7 (5.38%) GDPR
3 7 (5.38%) ISO/IEC 27001
System Software
1 15 (11.54%) Apache Flume
2 12 (9.23%) Docker
3 7 (5.38%) Apache ZooKeeper
4 1 (0.77%) Alluxio
4 1 (0.77%) Ceph
4 1 (0.77%) etcd
4 1 (0.77%) Gluster
Systems Management
1 39 (30.00%) Hadoop YARN
2 24 (18.46%) Oozie
3 11 (8.46%) Kubernetes
4 8 (6.15%) Ansible
5 7 (5.38%) Terraform
6 6 (4.62%) Ambari
6 6 (4.62%) Single Sign-On
7 5 (3.85%) Mesos
7 5 (3.85%) Puppet
8 3 (2.31%) Opscode Chef
9 2 (1.54%) Graphite
9 2 (1.54%) Kibana
9 2 (1.54%) Nagios
9 2 (1.54%) Opsview
10 1 (0.77%) Grafana
10 1 (0.77%) Istio
10 1 (0.77%) Knative
10 1 (0.77%) Prometheus
Vendors
1 32 (24.62%) Cloudera
2 13 (10.00%) Oracle
3 10 (7.69%) Hortonworks
4 9 (6.92%) Databricks
5 8 (6.15%) Google
6 5 (3.85%) Informatica
6 5 (3.85%) SolarWinds
6 5 (3.85%) Tableau
6 5 (3.85%) Talend
7 3 (2.31%) Adobe
7 3 (2.31%) MapR
7 3 (2.31%) Splunk
7 3 (2.31%) Teradata
8 2 (1.54%) IBM
9 1 (0.77%) Confluent
9 1 (0.77%) Microsoft