Period
to 23 October 2018

The following table provides summary statistics for permanent job vacancies with a requirement for Data Ingestion skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Ingestion over the 6 months to 23 October 2018 with a comparison to the same period in the previous 2 years.

Data Ingestion
UK
6 months to
23 Oct 2018
Same period 2017 Same period 2016
Rank 1029 1096 1191
Rank change year-on-year +67 +95 +112
Permanent jobs citing Data Ingestion 128 100 46
As % of all permanent IT jobs advertised in the UK 0.078% 0.057% 0.024%
As % of the Processes & Methodologies category 0.085% 0.062% 0.027%
Number of salaries quoted 88 80 31
UK median annual salary £68,750 £70,000 £57,500
Median salary % change year-on-year -1.79% +21.74% -
10th Percentile £44,775 £52,500 £42,500
90th Percentile £105,000 £80,000 £93,750
UK excluding London median annual salary £65,000 £70,000 £55,000
% change year-on-year -7.14% +27.27% -20.00%

Data Ingestion is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

All Process and Methodology Skills
UK
Permanent vacancies with a requirement for process or methodology skills 150,004 160,879 172,382
As % of all permanent IT jobs advertised in the UK 91.04% 91.57% 90.91%
Number of salaries quoted 119,274 126,930 143,463
UK median annual salary £50,000 £50,000 £48,500
Median salary % change year-on-year - +3.09% +2.11%
10th Percentile £29,000 £28,250 £27,500
90th Percentile £83,750 £80,000 £78,750
UK excluding London median annual salary £45,000 £42,500 £42,500
% change year-on-year +5.88% - -

Data Ingestion
Job Vacancy Trend

Job postings citing Data Ingestion as a percentage of all IT jobs advertised.

Job vacancy trend for Data Ingestion in the UK

Data Ingestion
Salary Trend

This chart provides the 3-month moving average for salaries quoted in permanent IT jobs citing Data Ingestion.

Salary trend for Data Ingestion in the UK

Data Ingestion
Salary Histogram

The salary distribution of IT jobs citing Data Ingestion over the 6 months to 23 October 2018.

Salary histogram for Data Ingestion in the UK

Data Ingestion
Top 10 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Ingestion within the UK over the 6 months to 23 October 2018. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +56 123 £68,750 -1.79% 27
London +47 79 £80,000 +25.49% 18
UK excluding London +36 49 £65,000 -7.14% 9
North of England - 20 £70,000 - 2
North West - 20 £70,000 - 2
South East +17 13 £57,500 - 4
West Midlands +18 7 £60,000 -14.29% 2
Midlands -6 7 £60,000 -14.29% 2
Scotland - 5 - -
East of England +31 4 £45,000 +5.88% 1

For the 6 months to 23 October 2018, IT jobs citing Data Ingestion also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Data Ingestion.

1 82 (64.06%) Python
2 81 (63.28%) Hadoop
3 78 (60.94%) Big Data
4 76 (59.38%) Apache Spark
5 74 (57.81%) Analytics
6 72 (56.25%) Java
7 69 (53.91%) Agile Software Development
7 69 (53.91%) Amazon AWS
8 54 (42.19%) Scala
9 51 (39.84%) Kafka
9 51 (39.84%) Data Warehouse
10 50 (39.06%) Data Engineering
10 50 (39.06%) NoSQL
10 50 (39.06%) Data Architecture
10 50 (39.06%) Apache Hive
11 47 (36.72%) Data Modelling
12 43 (33.59%) ETL
12 43 (33.59%) Microsoft Azure
13 40 (31.25%) Machine Learning
13 40 (31.25%) DevOps
14 39 (30.47%) HBase
15 38 (29.69%) Data Lake
16 37 (28.91%) Impala
16 37 (28.91%) Computer Science
17 36 (28.13%) Microsoft
17 36 (28.13%) Open Source
17 36 (28.13%) HDFS
18 34 (26.56%) MongoDB
18 34 (26.56%) Data Analytics
19 33 (25.78%) Apache Cassandra

Data Ingestion
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 76 (59.38%) Apache Spark
2 32 (25.00%) Apache
3 17 (13.28%) Apache Pig
4 13 (10.16%) Apache Solr
5 9 (7.03%) Elasticsearch
6 5 (3.91%) SAS
7 3 (2.34%) Oracle Workflow
7 3 (2.34%) Tomcat
7 3 (2.34%) WebLogic
8 1 (0.78%) Apache Airflow
Applications
1 2 (1.56%) Microsoft Excel
2 1 (0.78%) Microsoft PowerPoint
2 1 (0.78%) Microsoft Project
2 1 (0.78%) MS Visio
Business Applications
1 3 (2.34%) SAP HR
2 1 (0.78%) Eloqua
2 1 (0.78%) Oracle AML
Cloud Services
1 69 (53.91%) Amazon AWS
2 43 (33.59%) Microsoft Azure
3 23 (17.97%) Serverless
4 22 (17.19%) Amazon EMR
5 21 (16.41%) Amazon EC2
5 21 (16.41%) Cloud Computing
6 18 (14.06%) IaaS
6 18 (14.06%) PaaS
7 15 (11.72%) AWS Lambda
8 14 (10.94%) Google Cloud Platform
9 9 (7.03%) Cortana Intelligence Suite
9 9 (7.03%) SaaS
10 5 (3.91%) Azure Data Factory
11 4 (3.13%) GitHub
12 2 (1.56%) Amazon SQS
12 2 (1.56%) Azure Machine Learning
13 1 (0.78%) Amazon Kinesis
13 1 (0.78%) Data Lake Analytics
Communications & Networking
1 11 (8.59%) Internet
2 4 (3.13%) HTTP
Database & Business Intelligence
1 81 (63.28%) Hadoop
2 78 (60.94%) Big Data
3 51 (39.84%) Data Warehouse
4 50 (39.06%) Apache Hive
4 50 (39.06%) NoSQL
5 39 (30.47%) HBase
6 38 (29.69%) Data Lake
7 37 (28.91%) Impala
8 34 (26.56%) MongoDB
9 33 (25.78%) Apache Cassandra
10 27 (21.09%) Amazon Redshift
11 22 (17.19%) RDBMS
11 22 (17.19%) Relational Database
11 22 (17.19%) SQL Server
12 19 (14.84%) Flink
13 17 (13.28%) Amazon Athena
14 11 (8.59%) MySQL
14 11 (8.59%) Sqoop
15 9 (7.03%) Azure SQL Database
15 9 (7.03%) SQL Server Integration Services
Development Applications
1 22 (17.19%) JIRA
2 10 (7.81%) Git (software)
2 10 (7.81%) Jenkins
2 10 (7.81%) Visual Studio
3 8 (6.25%) Eclipse
3 8 (6.25%) ERwin
4 6 (4.69%) Subversion
5 5 (3.91%) Maven
5 5 (3.91%) Team Foundation Server
6 4 (3.13%) FindBugs
6 4 (3.13%) GitLab
6 4 (3.13%) IntelliJ
6 4 (3.13%) JUnit
7 2 (1.56%) TeamCity
General
1 23 (17.97%) Finance
2 11 (8.59%) Retail
3 10 (7.81%) Banking
4 9 (7.03%) Advertising
5 8 (6.25%) Marketing
6 7 (5.47%) Law
7 3 (2.34%) Investment Banking
8 2 (1.56%) Military
9 1 (0.78%) Front Office
9 1 (0.78%) Telecoms
Job Titles
1 51 (39.84%) Data Engineer
2 37 (28.91%) Architect
3 31 (24.22%) Big Data Engineer
4 27 (21.09%) Data Architect
5 23 (17.97%) Developer
6 15 (11.72%) Senior Architect
7 14 (10.94%) Senior Data Architect
8 13 (10.16%) Big Data Developer
8 13 (10.16%) Senior Data Engineer
9 10 (7.81%) Cloud Architect
9 10 (7.81%) Solutions Architect
10 9 (7.03%) AWS Architect
11 8 (6.25%) AWS Solutions Architect
11 8 (6.25%) Azure Architect
11 8 (6.25%) Cloud Solutions Architect
11 8 (6.25%) Consultant
11 8 (6.25%) Senior Developer
12 7 (5.47%) Analytics Consultant
12 7 (5.47%) Data Analytics Consultant
12 7 (5.47%) Data Consultant
Libraries, Frameworks & Software Standards
1 51 (39.84%) Kafka
2 22 (17.19%) Spark Streaming
3 21 (16.41%) Web Services
4 12 (9.38%) Elastic Stack
5 10 (7.81%) JSON
6 7 (5.47%) HTML
6 7 (5.47%) Natural Language Processing
7 6 (4.69%) J2EE
7 6 (4.69%) XML
8 5 (3.91%) Apache Avro
8 5 (3.91%) RESTful
9 4 (3.13%) AngularJS
9 4 (3.13%) CSS
9 4 (3.13%) HTML5
9 4 (3.13%) jQuery
9 4 (3.13%) React
9 4 (3.13%) Spring
10 3 (2.34%) JRuby
11 2 (1.56%) MLlib
11 2 (1.56%) scikit-learn
Miscellaneous
1 37 (28.91%) Computer Science
2 16 (12.50%) Analytical Skills
3 10 (7.81%) Internet of Things
4 9 (7.03%) Enterprise Software
5 8 (6.25%) Public Cloud
6 7 (5.47%) Reference Data
7 6 (4.69%) Smart City
8 5 (3.91%) Algorithms
9 4 (3.13%) Mainframe
9 4 (3.13%) Taxonomies
10 2 (1.56%) Cloud Native
11 1 (0.78%) Fintech
11 1 (0.78%) FMCG
11 1 (0.78%) Hybrid Cloud
11 1 (0.78%) Infotainment
11 1 (0.78%) Linux Command Line
11 1 (0.78%) Pricing Data
11 1 (0.78%) Private Cloud
11 1 (0.78%) Self-Motivation
11 1 (0.78%) Virtual Team
Operating Systems
1 16 (12.50%) Unix
2 8 (6.25%) Linux
3 5 (3.91%) Windows
4 1 (0.78%) AIX
4 1 (0.78%) CentOS
4 1 (0.78%) DC/OS
4 1 (0.78%) HPUX
4 1 (0.78%) Red Hat Enterprise Linux
4 1 (0.78%) Solaris
4 1 (0.78%) Windows Server
4 1 (0.78%) Windows Server 2008
Processes & Methodologies
1 74 (57.81%) Analytics
2 69 (53.91%) Agile Software Development
3 50 (39.06%) Data Architecture
3 50 (39.06%) Data Engineering
4 47 (36.72%) Data Modelling
5 43 (33.59%) ETL
6 40 (31.25%) DevOps
6 40 (31.25%) Machine Learning
7 36 (28.13%) Open Source
8 34 (26.56%) Data Analytics
9 33 (25.78%) Data Science
10 28 (21.88%) Mathematics
11 26 (20.31%) Configuration Management
12 20 (15.63%) MapReduce
13 19 (14.84%) Use Case
14 18 (14.06%) TDD
15 17 (13.28%) Change Management
15 17 (13.28%) Mentoring
15 17 (13.28%) Software Engineering
16 15 (11.72%) Microservices
Programming Languages
1 82 (64.06%) Python
2 72 (56.25%) Java
3 54 (42.19%) Scala
4 33 (25.78%) SQL
5 10 (7.81%) Shell Script
6 9 (7.03%) C++
7 8 (6.25%) C
7 8 (6.25%) Go
7 8 (6.25%) JavaScript
8 6 (4.69%) R
9 3 (2.34%) Clojure
9 3 (2.34%) Groovy
9 3 (2.34%) Jython
9 3 (2.34%) Kotlin
9 3 (2.34%) PL/SQL
10 2 (1.56%) HiveQL
10 2 (1.56%) MATLAB
11 1 (0.78%) C#
11 1 (0.78%) Perl
Qualifications
1 33 (25.78%) Degree
2 22 (17.19%) Computer Science Degree
3 18 (14.06%) Numerate Degree
4 8 (6.25%) AWS Certification
5 1 (0.78%) CISSP
5 1 (0.78%) GCIA
5 1 (0.78%) GCIH
5 1 (0.78%) PhD
5 1 (0.78%) PMI Certification
5 1 (0.78%) SANS
Quality Assurance & Compliance
1 13 (10.16%) QA
2 11 (8.59%) Data Quality
System Software
1 36 (28.13%) HDFS
2 18 (14.06%) Apache Flume
3 5 (3.91%) Apache ZooKeeper
4 4 (3.13%) Virtual Machines
5 2 (1.56%) Docker
Systems Management
1 18 (14.06%) Oozie
2 14 (10.94%) Hadoop YARN
3 11 (8.59%) Mesos
3 11 (8.59%) Terraform
4 10 (7.81%) Opscode Chef
4 10 (7.81%) Puppet
5 9 (7.03%) Kibana
6 4 (3.13%) Kubernetes
6 4 (3.13%) Rundeck
7 1 (0.78%) Ansible
7 1 (0.78%) Consul
7 1 (0.78%) Packer
Vendors
1 36 (28.13%) Microsoft
2 25 (19.53%) Oracle
3 22 (17.19%) Cloudera
4 20 (15.63%) Google
4 20 (15.63%) Talend
5 17 (13.28%) CA Agile Central
6 16 (12.50%) Cognizant
7 14 (10.94%) Tableau
8 10 (7.81%) Salesforce.com
9 9 (7.03%) Hortonworks
10 8 (6.25%) Databricks
11 7 (5.47%) SAP
12 4 (3.13%) Ab Initio
12 4 (3.13%) IBM
12 4 (3.13%) Informatica
13 3 (2.34%) MapR
14 1 (0.78%) ArcSight
14 1 (0.78%) Facebook
14 1 (0.78%) LogRhythm
14 1 (0.78%) Splunk