Period
to 27 October 2021

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 27 October 2021 with a comparison to the same period in the previous 2 years.

Data Pipelines
UK
6 months to
27 Oct 2021
Same period 2020 Same period 2019
Rank 258 344 436
Rank change year-on-year +86 +92 +285
Permanent jobs citing Data Pipeline 1,672 433 922
As % of all permanent jobs advertised in the UK 1.32% 0.87% 0.68%
As % of the Processes & Methodologies category 1.40% 0.94% 0.73%
Number of salaries quoted 975 353 725
10th Percentile £45,000 £45,000 £43,300
25th Percentile £52,500 £52,500 £52,500
Median annual salary (50th Percentile) £70,000 £65,000 £70,000
Median % change year-on-year +7.69% -7.14% +7.69%
75th Percentile £85,000 £76,250 £81,750
90th Percentile £95,000 £92,000 £95,000
UK excluding London median annual salary £57,500 £52,500 £55,000
% change year-on-year +9.52% -4.55% +4.76%

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

All Process and Methodology Skills
UK
Permanent vacancies with a requirement for process or methodology skills 119,650 46,099 125,961
As % of all permanent IT jobs advertised in the UK 94.21% 93.09% 92.60%
Number of salaries quoted 79,137 38,471 100,635
10th Percentile £32,750 £32,750 £30,000
25th Percentile £42,500 £42,500 £37,750
Median annual salary (50th Percentile) £57,500 £55,000 £52,500
Median % change year-on-year +4.55% +4.76% +5.00%
75th Percentile £77,500 £73,750 £70,500
90th Percentile £95,000 £90,000 £87,500
UK excluding London median annual salary £50,000 £49,000 £45,000
% change year-on-year +2.04% +8.89% -

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 27 October 2021.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 27 October 2021. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Job
Vacancies
England +79 1,508 £70,000 +7.69% 216
London +22 954 £75,000 +11.11% 147
UK excluding London +65 566 £57,500 +9.52% 75
Work from Home -44 335 £67,500 +8.00% 57
South East 0 195 £60,000 - 12
North of England -66 89 £62,500 +25.00% 22
Midlands -72 75 £52,500 +5.00% 13
South West -34 72 £47,500 +14.46% 12
East of England -59 72 £77,500 +19.23% 7
West Midlands -45 58 £55,000 +10.00% 11
Yorkshire -58 51 £62,500 +38.89% 4
Scotland -61 40 £52,000 +4.00% 5
North West -79 38 £62,500 +19.05% 17
East Midlands -53 17 £50,000 +5.26% 2
Wales -58 11 £50,000 +1.52% 4
Isle of Man +1 10 - -
Northern Ireland -39 3 - -

For the 6 months to 27 October 2021, IT jobs citing Data Pipeline also mentioned the following skills in order of popularity. The figures indicate the absolute number co-occurrences and as a proportion of all permanent job ads with a requirement for Data Pipeline.

1 1,199 (71.71%) Python
2 952 (56.94%) SQL
3 891 (53.29%) AWS
4 851 (50.90%) Analytics
5 599 (35.83%) ETL
6 567 (33.91%) Agile Software Development
7 559 (33.43%) Big Data
8 546 (32.66%) Data Science
9 535 (32.00%) Azure
10 485 (29.01%) Apache Spark
11 482 (28.83%) Data Warehouse
12 452 (27.03%) Business Intelligence
13 451 (26.97%) Data Modelling
13 451 (26.97%) DevOps
14 436 (26.08%) Data Engineering
15 422 (25.24%) Java
16 406 (24.28%) GCP
17 389 (23.27%) Software Engineering
18 380 (22.73%) Social Skills
19 346 (20.69%) Machine Learning
20 345 (20.63%) Kafka
21 327 (19.56%) NoSQL
22 310 (18.54%) Data Lake
23 287 (17.17%) Finance
24 286 (17.11%) Power Platform
25 281 (16.81%) Power BI
26 276 (16.51%) Visualisation
27 273 (16.33%) Docker
28 272 (16.27%) Relational Database
29 260 (15.55%) PostgreSQL

Data Pipeline
Co-occurring IT Skills by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same job type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 485 (29.01%) Apache Spark
2 196 (11.72%) Apache
3 177 (10.59%) Apache Airflow
4 65 (3.89%) Elasticsearch
5 42 (2.51%) GemFire
6 37 (2.21%) Confluence
7 26 (1.56%) Luigi
8 25 (1.50%) SAS
9 12 (0.72%) Apache Pulsar
10 11 (0.66%) Tomcat
11 10 (0.60%) Jupyter
12 9 (0.54%) Tornado
13 8 (0.48%) Apache Pig
13 8 (0.48%) Oracle Workflow
14 7 (0.42%) SharePoint
15 6 (0.36%) OpenStack
16 5 (0.30%) Ethereum
16 5 (0.30%) GlassFish
16 5 (0.30%) WebLogic
17 3 (0.18%) IIS
Applications
1 66 (3.95%) Microsoft Excel
2 10 (0.60%) Spreadsheet
3 9 (0.54%) SPSS
4 6 (0.36%) Microsoft Office
5 5 (0.30%) Adobe Illustrator
5 5 (0.30%) Fireworks
5 5 (0.30%) Photoshop
6 3 (0.18%) Microsoft PowerPoint
7 1 (0.060%) Chatbot
7 1 (0.060%) Figma
7 1 (0.060%) Sketch
Business Applications
1 12 (0.72%) Dynamics CRM
2 4 (0.24%) Summit
3 3 (0.18%) Eloqua
4 2 (0.12%) SAP S/4HANA
5 1 (0.060%) Markit EDM
5 1 (0.060%) Oracle Procure-to-Pay
5 1 (0.060%) Salesforce.com CRM
5 1 (0.060%) SAP R/3
Cloud Services
1 891 (53.29%) AWS
2 535 (32.00%) Azure
3 406 (24.28%) GCP
4 286 (17.11%) Power Platform
5 199 (11.90%) Snowflake
6 195 (11.66%) Serverless
7 164 (9.81%) Azure Data Factory
8 125 (7.48%) Amazon S3
9 122 (7.30%) AWS Lambda
9 122 (7.30%) GitHub
10 104 (6.22%) Amazon EMR
11 97 (5.80%) Amazon Kinesis
12 59 (3.53%) SaaS
13 54 (3.23%) Amazon EC2
14 52 (3.11%) Amazon SQS
15 51 (3.05%) Azure Synapse Analytics
15 51 (3.05%) Microsoft 365
16 50 (2.99%) Amazon ECS
17 44 (2.63%) AWS Data Pipeline
18 42 (2.51%) AWS CloudFormation
Communications & Networking
1 18 (1.08%) DNS
2 16 (0.96%) Internet
3 15 (0.90%) TCP/IP
4 9 (0.54%) DHCP
4 9 (0.54%) HTTPS
4 9 (0.54%) IPv4
4 9 (0.54%) IPv6
4 9 (0.54%) SMTP
4 9 (0.54%) SNMP
4 9 (0.54%) SSH
5 6 (0.36%) SMS
6 5 (0.30%) MQTT
7 4 (0.24%) Cisco ASA
7 4 (0.24%) Wireless
8 3 (0.18%) Firewall
8 3 (0.18%) FTP
8 3 (0.18%) Network Security
9 2 (0.12%) Broadband
9 2 (0.12%) HTTP
9 2 (0.12%) SSL
Database & Business Intelligence
1 559 (33.43%) Big Data
2 482 (28.83%) Data Warehouse
3 327 (19.56%) NoSQL
4 310 (18.54%) Data Lake
5 281 (16.81%) Power BI
6 272 (16.27%) Relational Database
7 260 (15.55%) PostgreSQL
8 247 (14.77%) Hadoop
9 209 (12.50%) Amazon Redshift
10 187 (11.18%) Tableau
11 169 (10.11%) BigQuery
12 134 (8.01%) MySQL
13 133 (7.95%) MongoDB
14 129 (7.72%) SQL Server
15 116 (6.94%) SQL Server Integration Services
16 108 (6.46%) Redis
17 101 (6.04%) Apache Cassandra
18 91 (5.44%) HBase
19 89 (5.32%) Metadata
20 85 (5.08%) Looker
Development Applications
1 236 (14.11%) Git (software)
2 121 (7.24%) Jenkins
3 70 (4.19%) JIRA
4 67 (4.01%) Cypress.io
5 66 (3.95%) Protractor
5 66 (3.95%) Storybook
6 48 (2.87%) Gradle
7 45 (2.69%) Cucumber
8 44 (2.63%) Sonatype Nexus
9 43 (2.57%) JUnit
10 42 (2.51%) Eclipse
10 42 (2.51%) IntelliJ
10 42 (2.51%) Mockito
11 34 (2.03%) CircleCI
12 28 (1.67%) Artifactory
13 21 (1.26%) Atlassian Bamboo
13 21 (1.26%) GitLab
13 21 (1.26%) Vagrant
14 17 (1.02%) git-flow
14 17 (1.02%) Terragrunt
General
1 380 (22.73%) Social Skills
2 287 (17.17%) Finance
3 238 (14.23%) Analytical Skills
4 139 (8.31%) Banking
5 119 (7.12%) Law
6 73 (4.37%) Inclusion and Diversity
7 72 (4.31%) Marketing
8 67 (4.01%) Retail
9 55 (3.29%) Games
10 37 (2.21%) Legal
10 37 (2.21%) Publishing
11 36 (2.15%) Public Sector
12 33 (1.97%) Manufacturing
13 23 (1.38%) Telecoms
14 21 (1.26%) Advertising
14 21 (1.26%) Pharmaceutical
15 14 (0.84%) Electronics
15 14 (0.84%) Organisational Skills
16 11 (0.66%) Czech Language
16 11 (0.66%) Presentation Skills
Job Titles
1 762 (45.57%) Data Engineer
2 215 (12.86%) Developer
3 140 (8.37%) Senior Data Engineer
4 127 (7.60%) Python Engineer
5 118 (7.06%) Software Engineer
6 113 (6.76%) Architect
7 93 (5.56%) Big Data Engineer
8 90 (5.38%) Analyst
9 78 (4.67%) Data Architect
10 71 (4.25%) Azure Engineer
11 64 (3.83%) Data Analyst
12 62 (3.71%) AWS Engineer
13 54 (3.23%) Data Scientist
13 54 (3.23%) Principal Engineer
14 46 (2.75%) Data Manager
15 45 (2.69%) Principal Software Engineer
16 44 (2.63%) Security Engineer
17 43 (2.57%) Python Developer
18 42 (2.51%) Cybersecurity Engineer
18 42 (2.51%) Software Developer
Libraries, Frameworks & Software Standards
1 345 (20.63%) Kafka
2 187 (11.18%) Node.js
3 163 (9.75%) React
4 117 (7.00%) CSS
5 112 (6.70%) Web Services
6 102 (6.10%) AngularJS
7 82 (4.90%) Spring
8 81 (4.84%) PySpark
8 81 (4.84%) Spark Streaming
9 74 (4.43%) Django
10 69 (4.13%) REST
11 67 (4.01%) Express
11 67 (4.01%) Redux
12 66 (3.95%) Apache Beam
12 66 (3.95%) Tailwind CSS
12 66 (3.95%) Twitter Bootstrap
13 65 (3.89%) HTML
14 61 (3.65%) JSON
14 61 (3.65%) RESTful
15 58 (3.47%) GraphQL
Miscellaneous
1 120 (7.18%) Algorithms
2 110 (6.58%) Data Structures
3 71 (4.25%) User Experience
4 65 (3.89%) Self-Motivation
5 60 (3.59%) Public Cloud
6 53 (3.17%) Greenfield Project
7 48 (2.87%) Linux Command Line
8 43 (2.57%) Management Information System
9 39 (2.33%) Cloud Native
10 38 (2.27%) Virtual Team
11 24 (1.44%) Social Media
12 23 (1.38%) Data Centre
13 22 (1.32%) Taxonomies
14 21 (1.26%) Mobile App
15 18 (1.08%) Mobile Games
16 17 (1.02%) Enterprise Software
16 17 (1.02%) Renewable Energy
17 16 (0.96%) Legacy Systems
18 14 (0.84%) Replication
19 13 (0.78%) Hybrid Cloud
Operating Systems
1 156 (9.33%) Linux
2 90 (5.38%) Unix
3 14 (0.84%) Windows
4 11 (0.66%) Apple iOS
5 8 (0.48%) Android
6 4 (0.24%) Mac OS
7 3 (0.18%) Windows Server
8 1 (0.060%) Alpine Linux
8 1 (0.060%) CentOS
8 1 (0.060%) Red Hat Enterprise Linux
8 1 (0.060%) Ubuntu
Processes & Methodologies
1 851 (50.90%) Analytics
2 599 (35.83%) ETL
3 567 (33.91%) Agile Software Development
4 546 (32.66%) Data Science
5 452 (27.03%) Business Intelligence
6 451 (26.97%) Data Modelling
6 451 (26.97%) DevOps
7 436 (26.08%) Data Engineering
8 389 (23.27%) Software Engineering
9 346 (20.69%) Machine Learning
10 276 (16.51%) Visualisation
11 247 (14.77%) Problem-Solving
12 238 (14.23%) Computer Science
13 225 (13.46%) Data Management
14 222 (13.28%) Artificial Intelligence
15 220 (13.16%) Microservices
16 185 (11.06%) Data Architecture
17 183 (10.94%) Data Analytics
18 173 (10.35%) Decision-Making
19 170 (10.17%) Continuous Integration
Programming Languages
1 1,199 (71.71%) Python
2 952 (56.94%) SQL
3 422 (25.24%) Java
4 172 (10.29%) Scala
5 150 (8.97%) JavaScript
6 140 (8.37%) C#
7 120 (7.18%) R
8 95 (5.68%) PowerShell
9 85 (5.08%) Shell Script
10 75 (4.49%) TypeScript
11 66 (3.95%) Go
12 54 (3.23%) T-SQL
13 48 (2.87%) Bash Shell
14 44 (2.63%) Groovy
15 40 (2.39%) Rust
16 36 (2.15%) C++
17 21 (1.26%) Ruby
18 17 (1.02%) C
19 13 (0.78%) VBA
20 11 (0.66%) PL/SQL
Qualifications
1 251 (15.01%) Degree
2 59 (3.53%) Computer Science Degree
3 42 (2.51%) OCP
3 42 (2.51%) Oracle Certification
4 21 (1.26%) Master's Degree
5 14 (0.84%) PhD
6 13 (0.78%) AWS Certification
7 11 (0.66%) Security Cleared
8 10 (0.60%) SC Cleared
9 6 (0.36%) Numerate Degree
10 4 (0.24%) Microsoft Certification
11 1 (0.060%) BPSS Clearance
11 1 (0.060%) DV Cleared
11 1 (0.060%) IBM Certification
Quality Assurance & Compliance
1 200 (11.96%) Data Quality
2 93 (5.56%) QA
3 42 (2.51%) GDPR
4 29 (1.73%) GRC
5 15 (0.90%) Accessibility
6 11 (0.66%) SLA
7 4 (0.24%) ISO/IEC 27001
7 4 (0.24%) PCI DSS
8 2 (0.12%) ISO 8583
8 2 (0.12%) MiFID
9 1 (0.060%) COBIT
9 1 (0.060%) NIST
System Software
1 273 (16.33%) Docker
2 50 (2.99%) HDFS
3 21 (1.26%) VirtualBox
4 7 (0.42%) Active Directory
5 5 (0.30%) Apache ZooKeeper
6 2 (0.12%) Apache Flume
7 1 (0.060%) Firmware
Systems Management
1 218 (13.04%) Kubernetes
2 182 (10.89%) Terraform
3 108 (6.46%) Ansible
4 93 (5.56%) Opscode Chef
5 23 (1.38%) Puppet
6 21 (1.26%) Docker Swarm
7 15 (0.90%) Hadoop YARN
8 8 (0.48%) Consul
9 7 (0.42%) Kubeflow
9 7 (0.42%) Oozie
10 6 (0.36%) Istio
11 3 (0.18%) Control-M
11 3 (0.18%) Grafana
12 2 (0.12%) Active Directory Federation Services
12 2 (0.12%) Ambari
12 2 (0.12%) FortiGate
12 2 (0.12%) Kibana
12 2 (0.12%) logstash
13 1 (0.060%) Mesos
13 1 (0.060%) Prometheus
Vendors
1 236 (14.11%) Google
2 227 (13.58%) Databricks
3 97 (5.80%) Microsoft
4 79 (4.72%) Oracle
5 70 (4.19%) Talend
6 66 (3.95%) Okta
7 53 (3.17%) Informatica
8 49 (2.93%) Palantir
9 45 (2.69%) Splunk
10 43 (2.57%) VMware
11 39 (2.33%) SAP
12 32 (1.91%) Hortonworks
13 31 (1.85%) Cloudera
14 24 (1.44%) IBM
15 19 (1.14%) Salesforce.com
16 18 (1.08%) Teradata
17 14 (0.84%) Cognos
18 12 (0.72%) Adobe
19 11 (0.66%) Dell
20 10 (0.60%) MKS