Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 27 April 2024 with a comparison to the same period in the previous 2 years.

6 months to
27 Apr 2024
Same period 2023 Same period 2022
Rank 229 279 228
Rank change year-on-year +50 -51 +82
Permanent jobs citing Data Pipeline 1,264 1,350 2,802
As % of all permanent jobs advertised in the UK 1.29% 1.29% 1.82%
As % of the Processes & Methodologies category 1.50% 1.35% 1.90%
Number of salaries quoted 695 817 1,112
10th Percentile £37,000 £47,000 £47,500
25th Percentile £49,875 £56,856 £57,375
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +15.38%
75th Percentile £83,750 £87,500 £87,500
90th Percentile £105,000 £100,000 £98,625
UK excluding London median annual salary £60,000 £60,000 £65,000
% change year-on-year - -7.69% +11.12%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 84,305 100,095 147,310
As % of all permanent jobs advertised in the UK 85.99% 95.64% 95.57%
Number of salaries quoted 59,591 58,745 83,129
10th Percentile £29,000 £34,000 £33,500
25th Percentile £40,000 £45,000 £43,750
Median annual salary (50th Percentile) £55,000 £62,000 £60,000
Median % change year-on-year -11.29% +3.33% +9.09%
75th Percentile £72,500 £82,500 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +9.38%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 27 April 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 27 April 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +35 1,019 £67,500 -2.80% 295
UK excluding London +50 780 £60,000 - 128
Work from Home -19 482 £67,500 - 153
London +38 428 £75,000 -6.25% 174
North of England +25 241 £51,500 -15.82% 33
Scotland +50 187 £67,500 +8.00% 10
North West +83 148 £50,000 - 18
South East +64 121 £67,500 +12.50% 23
Midlands +25 100 £51,500 -14.17% 24
South West -31 78 £67,500 +12.50% 14
West Midlands +12 57 £60,000 - 15
Yorkshire -40 49 £62,000 -7.56% 11
North East +38 44 £50,000 -22.78% 4
East Midlands +16 43 £47,500 -13.64% 9
East of England +32 32 £62,500 -1.96% 14
Wales +27 21 £52,500 -19.23% 9
Northern Ireland +27 3 £60,000 +9.09% 3

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 151 (11.95%) Apache Spark
2 110 (8.70%) Apache
3 76 (6.01%) Apache Airflow
4 6 (0.47%) SharePoint
5 5 (0.40%) SAS
6 3 (0.24%) Microsoft Exchange
7 2 (0.16%) Luigi
8 1 (0.079%) Apache Synapse
8 1 (0.079%) Confluence
Applications
1 23 (1.82%) Microsoft Excel
2 8 (0.63%) Microsoft Office
3 4 (0.32%) Spreadsheet
4 3 (0.24%) Microsoft PowerPoint
4 3 (0.24%) MS Visio
Business Applications
1 4 (0.32%) Primavera
2 1 (0.079%) Salesforce CRM
2 1 (0.079%) SAP S/4HANA
2 1 (0.079%) Shopify
Cloud Services
1 458 (36.23%) AWS
1 458 (36.23%) Azure
2 355 (28.09%) GCP
3 209 (16.53%) Power Platform
4 161 (12.74%) Azure Data Factory
5 153 (12.10%) Snowflake
6 85 (6.72%) Serverless
7 66 (5.22%) dbt
8 65 (5.14%) Amazon S3
9 63 (4.98%) Azure Synapse Analytics
10 59 (4.67%) Cloud Computing
11 57 (4.51%) Matillion
12 47 (3.72%) GitHub
13 42 (3.32%) Azure DevOps
14 38 (3.01%) AWS Lambda
15 37 (2.93%) Azure Cosmos DB
16 23 (1.82%) SaaS
17 22 (1.74%) Amazon EMR
17 22 (1.74%) GitHub Actions
17 22 (1.74%) PaaS
Communications & Networking
1 65 (5.14%) Kerberos
2 18 (1.42%) DSL
3 9 (0.71%) Firewall
4 2 (0.16%) Intrusion Detection
4 2 (0.16%) Network Security
4 2 (0.16%) VPN
5 1 (0.079%) 3GPP
5 1 (0.079%) 5G
5 1 (0.079%) PPP
5 1 (0.079%) SMS
Database & Business Intelligence
1 365 (28.88%) Data Warehouse
2 255 (20.17%) Data Lake
3 235 (18.59%) BigQuery
4 206 (16.30%) Power BI
5 193 (15.27%) Looker
6 177 (14.00%) Tableau
7 147 (11.63%) Big Data
8 140 (11.08%) SQL Server
9 130 (10.28%) Metadata
10 124 (9.81%) NoSQL
11 76 (6.01%) SQL Server Integration Services
12 65 (5.14%) Amazon Redshift
13 56 (4.43%) PostgreSQL
14 53 (4.19%) Hadoop
15 47 (3.72%) Elasticsearch
16 44 (3.48%) Azure SQL Data Warehouse
17 42 (3.32%) Azure SQL Database
17 42 (3.32%) Star Schema
18 38 (3.01%) Relational Database
19 35 (2.77%) Delta Lake
Development Applications
1 228 (18.04%) Git
2 123 (9.73%) Jenkins
3 71 (5.62%) Bitbucket
4 65 (5.14%) Artifactory
5 47 (3.72%) GitLab
6 15 (1.19%) Spinnaker
7 10 (0.79%) CircleCI
8 8 (0.63%) Oracle Forms
9 6 (0.47%) JIRA
9 6 (0.47%) RStudio
10 5 (0.40%) GDB
11 4 (0.32%) Octopus Deploy
11 4 (0.32%) TeamCity
11 4 (0.32%) Visual Studio Team System
12 3 (0.24%) Visual Studio
13 2 (0.16%) Cucumber
13 2 (0.16%) Selenium
13 2 (0.16%) SQL Server Management Studio (SSMS)
13 2 (0.16%) webpack
14 1 (0.079%) SpecFlow
General
1 403 (31.88%) Analytical Skills
2 312 (24.68%) Finance
3 302 (23.89%) Social Skills
4 195 (15.43%) Inclusion and Diversity
5 162 (12.82%) Law
6 128 (10.13%) Marketing
7 127 (10.05%) Presentation Skills
8 125 (9.89%) Retail
9 95 (7.52%) Banking
10 55 (4.35%) Legal
11 38 (3.01%) Public Sector
12 22 (1.74%) Games
13 21 (1.66%) Investment Banking
14 20 (1.58%) Financial Institution
15 15 (1.19%) Billing
16 13 (1.03%) Influencing Skills
17 11 (0.87%) Back Office
17 11 (0.87%) Retail Banking
18 10 (0.79%) Telecoms
19 9 (0.71%) Front Office
Job Titles
1 679 (53.72%) Data Engineer
2 314 (24.84%) Senior
3 180 (14.24%) Platform Engineer
4 177 (14.00%) Lead
5 176 (13.92%) Principal Engineer
6 173 (13.69%) Principal Data Engineer
7 136 (10.76%) Senior Data Engineer
8 129 (10.21%) Software Engineer
9 119 (9.41%) Analyst
10 108 (8.54%) Data Analyst
11 82 (6.49%) Developer
12 66 (5.22%) Lead Software Engineer
13 61 (4.83%) Architect
14 57 (4.51%) AWS Engineer
15 56 (4.43%) Data Architect
15 56 (4.43%) Senior Analyst
16 52 (4.11%) Senior Data Analyst
16 52 (4.11%) Senior Software Engineer
17 47 (3.72%) Azure Engineer
18 46 (3.64%) Lead Data Engineer
Libraries, Frameworks & Software Standards
1 110 (8.70%) Kafka
2 76 (6.01%) JSON
3 75 (5.93%) XML
4 73 (5.78%) RESTful
5 72 (5.70%) PySpark
6 69 (5.46%) Web Services
7 65 (5.14%) JSON-LD
7 65 (5.14%) LDAP
7 65 (5.14%) OpenID
7 65 (5.14%) SAML
8 53 (4.19%) Pandas
9 39 (3.09%) Apache Beam
10 23 (1.82%) Apache NiFi
10 23 (1.82%) XGBoost
11 19 (1.50%) PyTorch
11 19 (1.50%) TensorFlow
12 16 (1.27%) .NET
12 16 (1.27%) REST
13 15 (1.19%) React
14 12 (0.95%) Flask
Miscellaneous
1 213 (16.85%) Self-Motivation
2 101 (7.99%) Management Information System
3 100 (7.91%) Data Structures
4 85 (6.72%) Algorithms
5 73 (5.78%) Cloud Native
6 66 (5.22%) Reference Data
7 65 (5.14%) Enterprise Cloud
8 64 (5.06%) Onboarding
9 56 (4.43%) Product Ownership
10 33 (2.61%) Public Cloud
11 24 (1.90%) Taxonomies
12 23 (1.82%) IoT
13 20 (1.58%) Greenfield Project
14 16 (1.27%) Distributed Systems
15 14 (1.11%) Enterprise Software
16 11 (0.87%) Hedge funds
17 9 (0.71%) Client/Server
17 9 (0.71%) Distributed Applications
18 8 (0.63%) Analytical Mindset
18 8 (0.63%) Data Centre
Operating Systems
1 34 (2.69%) Linux
2 18 (1.42%) Unix
3 2 (0.16%) Windows
4 1 (0.079%) Android
4 1 (0.079%) Apple iOS
4 1 (0.079%) Windows Server
Processes & Methodologies
1 560 (44.30%) Agile
2 514 (40.66%) Data Engineering
3 484 (38.29%) ETL
4 463 (36.63%) Analytics
5 433 (34.26%) Data Science
5 433 (34.26%) Problem-Solving
6 414 (32.75%) Data Modelling
7 338 (26.74%) CI/CD
8 321 (25.40%) Visualisation
9 314 (24.84%) DevOps
10 288 (22.78%) Machine Learning
11 268 (21.20%) Business Intelligence
11 268 (21.20%) Data Analysis
12 254 (20.09%) Mentoring
13 224 (17.72%) Software Engineering
14 213 (16.85%) Platform Engineering
15 189 (14.95%) Artificial Intelligence
16 188 (14.87%) DataOps
17 164 (12.97%) Change Management
18 159 (12.58%) Data Transformation
Programming Languages
1 753 (59.57%) SQL
2 690 (54.59%) Python
3 150 (11.87%) Java
4 94 (7.44%) Scala
5 81 (6.41%) Go
6 65 (5.14%) Prolog
6 65 (5.14%) SPARQL
7 60 (4.75%) T-SQL
8 38 (3.01%) C#
9 28 (2.22%) R
10 22 (1.74%) PL/SQL
11 21 (1.66%) JavaScript
12 19 (1.50%) Data Analysis Expressions
13 13 (1.03%) Kotlin
13 13 (1.03%) TypeScript
14 9 (0.71%) C
14 9 (0.71%) PowerShell
15 5 (0.40%) MATLAB
16 4 (0.32%) C++
17 3 (0.24%) Clojure
Qualifications
1 230 (18.20%) Degree
2 115 (9.10%) DBS Check
3 66 (5.22%) Computer Science Degree
4 48 (3.80%) Master's Degree
5 29 (2.29%) Security Cleared
6 19 (1.50%) SC Cleared
7 18 (1.42%) Microsoft Certification
8 15 (1.19%) Azure Certification
9 14 (1.11%) DV Cleared
10 12 (0.95%) PhD
11 4 (0.32%) BPSS Clearance
11 4 (0.32%) CIMA
11 4 (0.32%) Numerate Degree
11 4 (0.32%) PMI Certification
12 3 (0.24%) AWS Certification
12 3 (0.24%) AWS Certified Solutions Architect
13 1 (0.079%) HND
13 1 (0.079%) MCSA
13 1 (0.079%) MCSE
Quality Assurance & Compliance
1 191 (15.11%) GDPR
2 160 (12.66%) Data Quality
3 19 (1.50%) QA
3 19 (1.50%) SLA
4 12 (0.95%) Accessibility
5 4 (0.32%) Solvency II
6 3 (0.24%) ISO/IEC 27001
7 2 (0.16%) HIPAA
8 1 (0.079%) California Consumer Privacy Act
8 1 (0.079%) GMP
8 1 (0.079%) NCSC
System Software
1 65 (5.14%) Active Directory
2 55 (4.35%) Docker
3 9 (0.71%) HDFS
Systems Management
1 96 (7.59%) Kubernetes
2 60 (4.75%) Terraform
3 8 (0.63%) Ansible
3 8 (0.63%) Argo
4 5 (0.40%) Kubeflow
Vendors
1 347 (27.45%) Databricks
2 237 (18.75%) Google
3 154 (12.18%) Microsoft
4 49 (3.88%) Oracle
5 29 (2.29%) Alteryx
6 20 (1.58%) Informatica
7 18 (1.42%) Cognos
7 18 (1.42%) Sybase
8 13 (1.03%) SAP
9 12 (0.95%) OpenAI
10 11 (0.87%) Cloudera
10 11 (0.87%) Confluent
11 10 (0.79%) Teradata
11 10 (0.79%) YouTube
12 7 (0.55%) Talend
13 6 (0.47%) Dataiku
13 6 (0.47%) F5
14 5 (0.40%) MuleSoft
15 3 (0.24%) SnapLogic
15 3 (0.24%) Twitter