Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 11 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
11 May 2024
Same period 2023 Same period 2022
Rank 231 284 223
Rank change year-on-year +53 -61 +79
Permanent jobs citing Data Pipeline 1,279 1,284 2,969
As % of all permanent jobs advertised in the UK 1.29% 1.27% 1.88%
As % of the Processes & Methodologies category 1.51% 1.33% 1.97%
Number of salaries quoted 690 766 1,087
10th Percentile £36,975 £46,250 £47,500
25th Percentile £48,750 £56,250 £57,500
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +15.38%
75th Percentile £84,688 £87,500 £87,500
90th Percentile £105,000 £100,000 £98,000
UK excluding London median annual salary £60,000 £60,000 £65,000
% change year-on-year - -7.69% +8.33%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 84,921 96,445 150,777
As % of all permanent jobs advertised in the UK 85.57% 95.59% 95.69%
Number of salaries quoted 59,902 56,808 82,456
10th Percentile £29,010 £34,000 £33,645
25th Percentile £40,000 £45,000 £43,750
Median annual salary (50th Percentile) £55,000 £61,000 £60,000
Median % change year-on-year -9.84% +1.67% +9.09%
75th Percentile £72,500 £81,250 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +10.53%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 11 May 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 11 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +31 1,036 £67,500 -2.80% 310
UK excluding London +39 756 £60,000 - 148
Work from Home -6 515 £67,500 - 164
London +53 466 £75,000 -6.25% 181
North of England +8 234 £51,500 -15.00% 35
Scotland +51 188 £67,000 +3.08% 15
North West +73 146 £48,750 -2.50% 18
South East +31 109 £67,500 +12.98% 26
Midlands +21 105 £51,500 -10.43% 25
South West -61 73 £67,500 +1.48% 22
West Midlands +1 55 £60,000 - 21
East Midlands +20 50 £43,750 -20.45% 4
Yorkshire -44 46 £59,750 -10.91% 14
North East +33 42 £52,500 -15.32% 3
East of England +36 29 £58,750 -7.84% 15
Wales +19 19 £46,131 -29.03% 6
Northern Ireland +24 3 £60,000 -7.69% 5

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 149 (11.65%) Apache Spark
2 102 (7.97%) Apache
3 81 (6.33%) Apache Airflow
4 7 (0.55%) SAS
5 5 (0.39%) SharePoint
6 3 (0.23%) Confluence
6 3 (0.23%) Luigi
6 3 (0.23%) Microsoft Exchange
7 1 (0.078%) Apache Synapse
Applications
1 23 (1.80%) Microsoft Excel
2 8 (0.63%) Microsoft Office
3 4 (0.31%) Spreadsheet
4 3 (0.23%) Microsoft PowerPoint
4 3 (0.23%) MS Visio
Business Applications
1 4 (0.31%) Primavera
2 1 (0.078%) Salesforce CRM
2 1 (0.078%) SAP S/4HANA
2 1 (0.078%) Shopify
Cloud Services
1 498 (38.94%) AWS
2 458 (35.81%) Azure
3 352 (27.52%) GCP
4 199 (15.56%) Power Platform
5 160 (12.51%) Azure Data Factory
6 151 (11.81%) Snowflake
7 84 (6.57%) Serverless
8 67 (5.24%) dbt
9 66 (5.16%) Azure Synapse Analytics
10 61 (4.77%) Amazon S3
11 60 (4.69%) Cloud Computing
12 56 (4.38%) Matillion
13 41 (3.21%) GitHub
14 40 (3.13%) Azure DevOps
15 38 (2.97%) AWS Lambda
16 36 (2.81%) Azure Cosmos DB
17 26 (2.03%) Amazon EMR
18 23 (1.80%) SaaS
19 20 (1.56%) AWS Step Functions
19 20 (1.56%) PaaS
Communications & Networking
1 65 (5.08%) Kerberos
2 18 (1.41%) DSL
3 8 (0.63%) Firewall
4 2 (0.16%) Intrusion Detection
4 2 (0.16%) Network Security
4 2 (0.16%) VPN
5 1 (0.078%) 3GPP
5 1 (0.078%) 5G
5 1 (0.078%) PPP
Database & Business Intelligence
1 363 (28.38%) Data Warehouse
2 246 (19.23%) Data Lake
3 230 (17.98%) BigQuery
4 199 (15.56%) Tableau
5 195 (15.25%) Power BI
6 186 (14.54%) Looker
7 142 (11.10%) Big Data
8 141 (11.02%) SQL Server
9 130 (10.16%) Metadata
10 122 (9.54%) NoSQL
11 74 (5.79%) SQL Server Integration Services
12 64 (5.00%) Amazon Redshift
13 58 (4.53%) PostgreSQL
14 51 (3.99%) Hadoop
15 49 (3.83%) Elasticsearch
16 45 (3.52%) Azure SQL Data Warehouse
17 44 (3.44%) Azure SQL Database
18 42 (3.28%) Star Schema
19 38 (2.97%) Relational Database
20 33 (2.58%) Delta Lake
Development Applications
1 268 (20.95%) Git
2 113 (8.84%) Jenkins
3 71 (5.55%) Bitbucket
4 65 (5.08%) Artifactory
5 50 (3.91%) GitLab
6 16 (1.25%) Spinnaker
7 10 (0.78%) CircleCI
8 9 (0.70%) JIRA
9 8 (0.63%) Oracle Forms
10 6 (0.47%) RStudio
11 5 (0.39%) GDB
12 4 (0.31%) Octopus Deploy
13 3 (0.23%) TeamCity
13 3 (0.23%) Visual Studio
14 2 (0.16%) Visual Studio Team System
14 2 (0.16%) webpack
15 1 (0.078%) Cucumber
15 1 (0.078%) ERwin
15 1 (0.078%) Mockito
15 1 (0.078%) SpecFlow
General
1 434 (33.93%) Analytical Skills
2 333 (26.04%) Social Skills
3 308 (24.08%) Finance
4 206 (16.11%) Inclusion and Diversity
5 171 (13.37%) Law
6 155 (12.12%) Presentation Skills
7 146 (11.42%) Retail
8 127 (9.93%) Marketing
9 81 (6.33%) Banking
10 55 (4.30%) Legal
11 37 (2.89%) Public Sector
12 22 (1.72%) Games
13 21 (1.64%) Investment Banking
14 20 (1.56%) Financial Institution
15 16 (1.25%) Billing
16 12 (0.94%) Back Office
16 12 (0.94%) Influencing Skills
17 11 (0.86%) Front Office
17 11 (0.86%) Retail Banking
17 11 (0.86%) Telecoms
Job Titles
1 676 (52.85%) Data Engineer
2 338 (26.43%) Senior
3 179 (14.00%) Platform Engineer
4 178 (13.92%) Principal Engineer
5 175 (13.68%) Lead
5 175 (13.68%) Principal Data Engineer
6 146 (11.42%) Analyst
7 136 (10.63%) Data Analyst
8 135 (10.56%) Software Engineer
9 127 (9.93%) Senior Data Engineer
10 84 (6.57%) Senior Analyst
11 82 (6.41%) Senior Data Analyst
12 81 (6.33%) Developer
13 69 (5.39%) Lead Software Engineer
14 59 (4.61%) Architect
15 56 (4.38%) Senior Software Engineer
16 55 (4.30%) AWS Engineer
17 54 (4.22%) Data Architect
18 49 (3.83%) Lead Data Engineer
19 47 (3.67%) Azure Engineer
Libraries, Frameworks & Software Standards
1 95 (7.43%) Kafka
2 78 (6.10%) PySpark
3 74 (5.79%) JSON
3 74 (5.79%) XML
4 73 (5.71%) RESTful
5 70 (5.47%) Web Services
6 65 (5.08%) JSON-LD
6 65 (5.08%) LDAP
6 65 (5.08%) OpenID
6 65 (5.08%) SAML
7 50 (3.91%) Pandas
8 36 (2.81%) Apache Beam
9 23 (1.80%) Apache NiFi
9 23 (1.80%) React
9 23 (1.80%) XGBoost
10 20 (1.56%) .NET
10 20 (1.56%) PyTorch
10 20 (1.56%) TensorFlow
11 16 (1.25%) REST
12 13 (1.02%) .NET Core
Miscellaneous
1 213 (16.65%) Self-Motivation
2 104 (8.13%) Data Structures
3 101 (7.90%) Management Information System
4 85 (6.65%) Algorithms
5 76 (5.94%) Cloud Native
6 65 (5.08%) Enterprise Cloud
6 65 (5.08%) Onboarding
6 65 (5.08%) Reference Data
7 52 (4.07%) Product Ownership
8 29 (2.27%) Public Cloud
9 24 (1.88%) Taxonomies
10 22 (1.72%) IoT
11 18 (1.41%) Greenfield Project
12 17 (1.33%) Hedge funds
13 14 (1.09%) Distributed Systems
14 13 (1.02%) Enterprise Software
15 9 (0.70%) Blog
15 9 (0.70%) Client/Server
15 9 (0.70%) Distributed Applications
15 9 (0.70%) Private Cloud
Operating Systems
1 40 (3.13%) Linux
2 20 (1.56%) Unix
3 2 (0.16%) Windows
4 1 (0.078%) Android
4 1 (0.078%) Apple iOS
4 1 (0.078%) Windows Server
Processes & Methodologies
1 549 (42.92%) Agile
2 510 (39.87%) Data Engineering
3 491 (38.39%) Analytics
4 468 (36.59%) ETL
5 466 (36.43%) Problem-Solving
6 427 (33.39%) Data Science
7 412 (32.21%) Data Modelling
8 347 (27.13%) Visualisation
9 334 (26.11%) CI/CD
10 301 (23.53%) Data Analysis
11 292 (22.83%) DevOps
12 273 (21.34%) Machine Learning
13 260 (20.33%) Business Intelligence
14 257 (20.09%) Mentoring
15 238 (18.61%) Software Engineering
16 218 (17.04%) Platform Engineering
17 189 (14.78%) DataOps
18 183 (14.31%) Artificial Intelligence
19 165 (12.90%) Change Management
20 163 (12.74%) Data Transformation
Programming Languages
1 801 (62.63%) SQL
2 735 (57.47%) Python
3 168 (13.14%) Java
4 96 (7.51%) Scala
5 83 (6.49%) Go
6 65 (5.08%) Prolog
6 65 (5.08%) SPARQL
7 60 (4.69%) T-SQL
8 39 (3.05%) C#
9 30 (2.35%) R
10 25 (1.95%) JavaScript
11 22 (1.72%) PL/SQL
11 22 (1.72%) TypeScript
12 17 (1.33%) Kotlin
13 16 (1.25%) Data Analysis Expressions
14 9 (0.70%) C
14 9 (0.70%) PowerShell
15 6 (0.47%) Clojure
16 5 (0.39%) MATLAB
17 4 (0.31%) C++
Qualifications
1 263 (20.56%) Degree
2 151 (11.81%) DBS Check
3 65 (5.08%) Computer Science Degree
4 44 (3.44%) Master's Degree
5 30 (2.35%) Security Cleared
6 21 (1.64%) SC Cleared
7 16 (1.25%) Microsoft Certification
8 14 (1.09%) Azure Certification
8 14 (1.09%) DV Cleared
9 12 (0.94%) PhD
10 5 (0.39%) Numerate Degree
11 4 (0.31%) CIMA
11 4 (0.31%) PMI Certification
12 3 (0.23%) AWS Certification
12 3 (0.23%) AWS Certified Solutions Architect
12 3 (0.23%) BPSS Clearance
13 1 (0.078%) HND
13 1 (0.078%) Postgraduate
Quality Assurance & Compliance
1 186 (14.54%) GDPR
2 174 (13.60%) Data Quality
3 23 (1.80%) QA
4 19 (1.49%) SLA
5 11 (0.86%) Accessibility
6 4 (0.31%) Solvency II
7 3 (0.23%) ISO/IEC 27001
8 1 (0.078%) California Consumer Privacy Act
8 1 (0.078%) GMP
8 1 (0.078%) HIPAA
8 1 (0.078%) NCSC
System Software
1 67 (5.24%) Active Directory
2 52 (4.07%) Docker
3 9 (0.70%) HDFS
4 2 (0.16%) Virtual Machines
Systems Management
1 103 (8.05%) Kubernetes
2 54 (4.22%) Terraform
3 11 (0.86%) Ansible
4 8 (0.63%) Argo
5 5 (0.39%) Kubeflow
6 1 (0.078%) Grafana
6 1 (0.078%) Prometheus
Vendors
1 374 (29.24%) Databricks
2 231 (18.06%) Google
3 148 (11.57%) Microsoft
4 52 (4.07%) Oracle
5 29 (2.27%) Alteryx
6 19 (1.49%) Informatica
7 18 (1.41%) Cognos
7 18 (1.41%) Sybase
8 12 (0.94%) OpenAI
9 11 (0.86%) Cloudera
10 10 (0.78%) SAP
10 10 (0.78%) Teradata
10 10 (0.78%) YouTube
11 7 (0.55%) Confluent
12 6 (0.47%) Dataiku
12 6 (0.47%) F5
12 6 (0.47%) Talend
13 5 (0.39%) Bloomberg
14 3 (0.23%) Splunk
14 3 (0.23%) Twitter