Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 25 April 2024 with a comparison to the same period in the previous 2 years.

6 months to
25 Apr 2024
Same period 2023 Same period 2022
Rank 235 278 223
Rank change year-on-year +43 -55 +85
Permanent jobs citing Data Pipeline 1,259 1,368 2,810
As % of all permanent jobs advertised in the UK 1.28% 1.30% 1.83%
As % of the Processes & Methodologies category 1.49% 1.36% 1.91%
Number of salaries quoted 698 825 1,110
10th Percentile £37,000 £47,000 £47,500
25th Percentile £50,000 £57,046 £57,500
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +15.38%
75th Percentile £83,563 £87,500 £87,500
90th Percentile £105,000 £100,000 £97,500
UK excluding London median annual salary £60,000 £60,000 £65,000
% change year-on-year - -7.69% +11.12%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 84,476 100,576 147,122
As % of all permanent jobs advertised in the UK 86.05% 95.63% 95.59%
Number of salaries quoted 59,789 58,947 82,930
10th Percentile £29,000 £34,000 £33,476
25th Percentile £40,000 £45,000 £43,750
Median annual salary (50th Percentile) £55,000 £61,700 £60,000
Median % change year-on-year -10.86% +2.83% +9.09%
75th Percentile £72,500 £82,500 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +9.38%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 25 April 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 25 April 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +35 1,014 £67,500 -2.80% 286
UK excluding London +45 783 £60,000 - 119
Work from Home -19 476 £69,235 +2.57% 152
London +31 420 £75,000 -6.25% 176
North of England +8 243 £52,500 -14.19% 32
Scotland +57 187 £67,500 +8.00% 10
North West +83 148 £50,000 - 18
South East +53 121 £67,500 +12.50% 20
Midlands +22 99 £52,000 -13.33% 25
South West -52 80 £67,500 +12.50% 10
West Midlands +7 59 £60,000 - 15
Yorkshire -44 51 £66,000 -1.60% 10
North East +40 44 £50,000 -22.78% 4
East Midlands +12 40 £47,500 -13.64% 10
East of England +46 32 £62,500 -3.85% 13
Wales +19 21 £52,500 -19.23% 9
Northern Ireland +17 3 £60,000 +9.09% 2

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 155 (12.31%) Apache Spark
2 114 (9.05%) Apache
3 76 (6.04%) Apache Airflow
4 6 (0.48%) SharePoint
5 5 (0.40%) SAS
6 3 (0.24%) Microsoft Exchange
7 2 (0.16%) Luigi
8 1 (0.079%) Apache Synapse
8 1 (0.079%) Confluence
Applications
1 23 (1.83%) Microsoft Excel
2 8 (0.64%) Microsoft Office
3 4 (0.32%) Spreadsheet
4 3 (0.24%) Microsoft PowerPoint
4 3 (0.24%) MS Visio
Business Applications
1 4 (0.32%) Primavera
2 1 (0.079%) Salesforce CRM
2 1 (0.079%) SAP S/4HANA
2 1 (0.079%) Shopify
Cloud Services
1 459 (36.46%) Azure
2 449 (35.66%) AWS
3 361 (28.67%) GCP
4 215 (17.08%) Power Platform
5 160 (12.71%) Azure Data Factory
6 154 (12.23%) Snowflake
7 87 (6.91%) Serverless
8 68 (5.40%) Amazon S3
9 66 (5.24%) dbt
10 62 (4.92%) Azure Synapse Analytics
11 59 (4.69%) Cloud Computing
12 58 (4.61%) Matillion
13 53 (4.21%) GitHub
14 42 (3.34%) Azure DevOps
15 39 (3.10%) AWS Lambda
16 37 (2.94%) Azure Cosmos DB
17 26 (2.07%) GitHub Actions
18 23 (1.83%) SaaS
19 22 (1.75%) Amazon EMR
19 22 (1.75%) PaaS
Communications & Networking
1 65 (5.16%) Kerberos
2 18 (1.43%) DSL
3 10 (0.79%) Firewall
4 2 (0.16%) Intrusion Detection
4 2 (0.16%) Network Security
4 2 (0.16%) VPN
5 1 (0.079%) 3GPP
5 1 (0.079%) 5G
5 1 (0.079%) PPP
5 1 (0.079%) SMS
Database & Business Intelligence
1 364 (28.91%) Data Warehouse
2 259 (20.57%) Data Lake
3 239 (18.98%) BigQuery
4 212 (16.84%) Power BI
5 197 (15.65%) Looker
6 169 (13.42%) Tableau
7 150 (11.91%) Big Data
8 139 (11.04%) SQL Server
9 131 (10.41%) Metadata
10 128 (10.17%) NoSQL
11 76 (6.04%) SQL Server Integration Services
12 68 (5.40%) Amazon Redshift
13 60 (4.77%) PostgreSQL
14 53 (4.21%) Hadoop
15 47 (3.73%) Elasticsearch
16 44 (3.49%) Azure SQL Data Warehouse
17 42 (3.34%) Azure SQL Database
17 42 (3.34%) Relational Database
17 42 (3.34%) Star Schema
18 35 (2.78%) MySQL
Development Applications
1 216 (17.16%) Git
2 129 (10.25%) Jenkins
3 71 (5.64%) Bitbucket
4 65 (5.16%) Artifactory
5 47 (3.73%) GitLab
6 16 (1.27%) Spinnaker
7 10 (0.79%) CircleCI
8 8 (0.64%) Oracle Forms
9 7 (0.56%) JIRA
10 6 (0.48%) RStudio
11 5 (0.40%) GDB
12 4 (0.32%) Octopus Deploy
12 4 (0.32%) TeamCity
12 4 (0.32%) Visual Studio Team System
13 3 (0.24%) Visual Studio
14 2 (0.16%) Cucumber
14 2 (0.16%) Selenium
14 2 (0.16%) SQL Server Management Studio (SSMS)
14 2 (0.16%) webpack
15 1 (0.079%) SpecFlow
General
1 396 (31.45%) Analytical Skills
2 318 (25.26%) Finance
3 290 (23.03%) Social Skills
4 195 (15.49%) Inclusion and Diversity
5 162 (12.87%) Law
6 128 (10.17%) Marketing
7 118 (9.37%) Retail
8 115 (9.13%) Presentation Skills
9 100 (7.94%) Banking
10 54 (4.29%) Legal
11 38 (3.02%) Public Sector
12 22 (1.75%) Games
13 21 (1.67%) Investment Banking
14 20 (1.59%) Financial Institution
15 15 (1.19%) Billing
15 15 (1.19%) Influencing Skills
16 11 (0.87%) Back Office
16 11 (0.87%) Retail Banking
17 10 (0.79%) Telecoms
18 9 (0.71%) Front Office
Job Titles
1 677 (53.77%) Data Engineer
2 302 (23.99%) Senior
3 180 (14.30%) Platform Engineer
4 179 (14.22%) Lead
5 176 (13.98%) Principal Engineer
6 173 (13.74%) Principal Data Engineer
7 136 (10.80%) Senior Data Engineer
8 130 (10.33%) Software Engineer
9 107 (8.50%) Analyst
10 96 (7.63%) Data Analyst
11 82 (6.51%) Developer
12 66 (5.24%) Lead Software Engineer
13 63 (5.00%) Architect
14 58 (4.61%) Data Architect
15 57 (4.53%) AWS Engineer
16 52 (4.13%) Senior Software Engineer
17 47 (3.73%) Azure Engineer
18 46 (3.65%) Lead Data Engineer
19 44 (3.49%) Senior Analyst
20 42 (3.34%) Azure Data Engineer
Libraries, Frameworks & Software Standards
1 115 (9.13%) Kafka
2 76 (6.04%) JSON
3 75 (5.96%) XML
4 74 (5.88%) RESTful
5 71 (5.64%) PySpark
6 69 (5.48%) Web Services
7 65 (5.16%) JSON-LD
7 65 (5.16%) LDAP
7 65 (5.16%) OpenID
7 65 (5.16%) SAML
8 53 (4.21%) Pandas
9 39 (3.10%) Apache Beam
10 23 (1.83%) Apache NiFi
10 23 (1.83%) XGBoost
11 19 (1.51%) PyTorch
11 19 (1.51%) TensorFlow
12 16 (1.27%) .NET
12 16 (1.27%) REST
13 15 (1.19%) React
14 12 (0.95%) Flask
Miscellaneous
1 213 (16.92%) Self-Motivation
2 102 (8.10%) Management Information System
3 99 (7.86%) Data Structures
4 85 (6.75%) Algorithms
5 73 (5.80%) Cloud Native
6 66 (5.24%) Reference Data
7 65 (5.16%) Enterprise Cloud
8 64 (5.08%) Onboarding
9 56 (4.45%) Product Ownership
10 35 (2.78%) Public Cloud
11 24 (1.91%) Taxonomies
12 23 (1.83%) IoT
13 20 (1.59%) Greenfield Project
14 16 (1.27%) Distributed Systems
14 16 (1.27%) Enterprise Software
15 11 (0.87%) Hedge funds
16 9 (0.71%) Client/Server
16 9 (0.71%) Distributed Applications
17 8 (0.64%) Analytical Mindset
17 8 (0.64%) Data Centre
Operating Systems
1 33 (2.62%) Linux
2 18 (1.43%) Unix
3 2 (0.16%) Windows
4 1 (0.079%) Android
4 1 (0.079%) Apple iOS
4 1 (0.079%) Windows Server
Processes & Methodologies
1 566 (44.96%) Agile
2 516 (40.98%) Data Engineering
3 490 (38.92%) ETL
4 454 (36.06%) Analytics
5 434 (34.47%) Data Science
6 422 (33.52%) Problem-Solving
7 415 (32.96%) Data Modelling
8 343 (27.24%) CI/CD
9 316 (25.10%) Visualisation
10 315 (25.02%) DevOps
11 295 (23.43%) Machine Learning
12 274 (21.76%) Business Intelligence
13 257 (20.41%) Data Analysis
14 253 (20.10%) Mentoring
15 225 (17.87%) Software Engineering
16 213 (16.92%) Platform Engineering
17 193 (15.33%) Artificial Intelligence
18 188 (14.93%) DataOps
19 164 (13.03%) Change Management
20 158 (12.55%) Data Transformation
Programming Languages
1 741 (58.86%) SQL
2 680 (54.01%) Python
3 152 (12.07%) Java
4 94 (7.47%) Scala
5 81 (6.43%) Go
6 65 (5.16%) Prolog
6 65 (5.16%) SPARQL
7 61 (4.85%) T-SQL
8 38 (3.02%) C#
9 28 (2.22%) R
10 22 (1.75%) PL/SQL
11 21 (1.67%) JavaScript
12 19 (1.51%) Data Analysis Expressions
13 14 (1.11%) TypeScript
14 13 (1.03%) Kotlin
15 9 (0.71%) C
15 9 (0.71%) PowerShell
16 5 (0.40%) MATLAB
17 4 (0.32%) C++
18 3 (0.24%) Clojure
Qualifications
1 219 (17.39%) Degree
2 102 (8.10%) DBS Check
3 66 (5.24%) Computer Science Degree
4 48 (3.81%) Master's Degree
5 28 (2.22%) Security Cleared
6 18 (1.43%) Microsoft Certification
6 18 (1.43%) SC Cleared
7 15 (1.19%) Azure Certification
8 13 (1.03%) DV Cleared
9 12 (0.95%) PhD
10 4 (0.32%) AWS Certification
10 4 (0.32%) BPSS Clearance
10 4 (0.32%) CIMA
10 4 (0.32%) Numerate Degree
10 4 (0.32%) PMI Certification
11 3 (0.24%) AWS Certified Solutions Architect
12 1 (0.079%) HND
12 1 (0.079%) MCSA
12 1 (0.079%) MCSE
Quality Assurance & Compliance
1 192 (15.25%) GDPR
2 158 (12.55%) Data Quality
3 20 (1.59%) QA
4 19 (1.51%) SLA
5 12 (0.95%) Accessibility
6 4 (0.32%) Solvency II
7 3 (0.24%) ISO/IEC 27001
8 2 (0.16%) HIPAA
9 1 (0.079%) California Consumer Privacy Act
9 1 (0.079%) GMP
9 1 (0.079%) NCSC
System Software
1 65 (5.16%) Active Directory
2 55 (4.37%) Docker
3 9 (0.71%) HDFS
Systems Management
1 97 (7.70%) Kubernetes
2 65 (5.16%) Terraform
3 8 (0.64%) Ansible
3 8 (0.64%) Argo
4 5 (0.40%) Kubeflow
5 1 (0.079%) Grafana
Vendors
1 335 (26.61%) Databricks
2 241 (19.14%) Google
3 154 (12.23%) Microsoft
4 49 (3.89%) Oracle
5 29 (2.30%) Alteryx
6 20 (1.59%) Informatica
7 18 (1.43%) Cognos
7 18 (1.43%) Sybase
8 15 (1.19%) SAP
9 12 (0.95%) Confluent
9 12 (0.95%) OpenAI
10 11 (0.87%) Cloudera
11 10 (0.79%) Teradata
11 10 (0.79%) YouTube
12 7 (0.56%) Talend
13 6 (0.48%) Dataiku
13 6 (0.48%) F5
14 5 (0.40%) MuleSoft
15 3 (0.24%) Splunk
15 3 (0.24%) Twitter