Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 28 May 2024 with a comparison to the same period in the previous 2 years.

6 months to
28 May 2024
Same period 2023 Same period 2022
Rank 231 265 212
Rank change year-on-year +34 -53 +87
Permanent jobs citing Data Pipeline 1,293 1,290 3,280
As % of all permanent jobs advertised in the UK 1.26% 1.33% 1.96%
As % of the Processes & Methodologies category 1.50% 1.39% 2.05%
Number of salaries quoted 685 707 1,068
10th Percentile £37,000 £46,250 £46,250
25th Percentile £50,000 £56,381 £57,500
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +11.11%
75th Percentile £85,000 £87,500 £87,500
90th Percentile £105,000 £100,000 £97,875
UK excluding London median annual salary £60,000 £60,000 £65,000
% change year-on-year - -7.69% +11.12%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 86,445 93,025 160,114
As % of all permanent jobs advertised in the UK 84.24% 95.56% 95.90%
Number of salaries quoted 60,550 54,985 83,280
10th Percentile £29,250 £34,000 £33,500
25th Percentile £40,000 £45,000 £43,750
Median annual salary (50th Percentile) £55,000 £61,000 £60,000
Median % change year-on-year -9.84% +1.67% +9.09%
75th Percentile £72,500 £81,250 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +9.38%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 28 May 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 28 May 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +36 1,058 £67,500 -3.57% 348
UK excluding London +30 745 £60,000 - 169
Work from Home +11 537 £67,500 -1.69% 161
London +71 497 £75,000 -6.25% 205
North of England +15 229 £51,500 -19.22% 52
Scotland +41 183 £67,000 +3.08% 21
North West +64 143 £47,500 -5.00% 30
Midlands +26 110 £52,250 -9.13% 25
South East +25 105 £67,500 +15.40% 29
South West -66 72 £67,500 -1.69% 20
West Midlands +12 59 £60,000 - 19
East Midlands +15 51 £40,000 -27.27% 6
Yorkshire -50 45 £62,000 -10.32% 17
North East +32 41 £52,500 +34.62% 5
East of England +13 30 £57,500 -9.80% 12
Wales +6 17 £46,131 -14.97% 4
Northern Ireland +8 3 £60,000 -7.69% 7

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 152 (11.76%) Apache Spark
2 102 (7.89%) Apache
3 76 (5.88%) Apache Airflow
4 11 (0.85%) SAS
5 5 (0.39%) Luigi
5 5 (0.39%) SharePoint
6 4 (0.31%) Confluence
7 3 (0.23%) Microsoft Exchange
8 1 (0.077%) Apache Pulsar
8 1 (0.077%) Apache Synapse
Applications
1 23 (1.78%) Microsoft Excel
2 9 (0.70%) Microsoft Office
3 4 (0.31%) Spreadsheet
4 3 (0.23%) Microsoft PowerPoint
4 3 (0.23%) MS Visio
Business Applications
1 4 (0.31%) Primavera
2 1 (0.077%) Salesforce CRM
2 1 (0.077%) SAP S/4HANA
2 1 (0.077%) Shopify
Cloud Services
1 517 (39.98%) AWS
2 457 (35.34%) Azure
3 346 (26.76%) GCP
4 198 (15.31%) Power Platform
5 160 (12.37%) Azure Data Factory
6 153 (11.83%) Snowflake
7 84 (6.50%) Serverless
8 73 (5.65%) Azure Synapse Analytics
9 64 (4.95%) dbt
10 63 (4.87%) Amazon S3
11 60 (4.64%) Cloud Computing
12 55 (4.25%) Matillion
13 40 (3.09%) AWS Lambda
14 37 (2.86%) GitHub
15 36 (2.78%) Azure Cosmos DB
16 35 (2.71%) Azure DevOps
17 28 (2.17%) Amazon EMR
18 23 (1.78%) SaaS
19 21 (1.62%) AWS Step Functions
20 19 (1.47%) Fivetran
Communications & Networking
1 65 (5.03%) Kerberos
2 17 (1.31%) DSL
3 6 (0.46%) Firewall
4 2 (0.15%) Intrusion Detection
4 2 (0.15%) Network Security
4 2 (0.15%) VPN
5 1 (0.077%) 3GPP
5 1 (0.077%) 5G
5 1 (0.077%) PPP
Database & Business Intelligence
1 354 (27.38%) Data Warehouse
2 241 (18.64%) Data Lake
3 227 (17.56%) BigQuery
4 219 (16.94%) Tableau
5 194 (15.00%) Power BI
6 186 (14.39%) Looker
7 143 (11.06%) SQL Server
8 142 (10.98%) Big Data
9 126 (9.74%) NoSQL
10 123 (9.51%) Metadata
11 72 (5.57%) SQL Server Integration Services
12 65 (5.03%) Amazon Redshift
13 57 (4.41%) PostgreSQL
14 51 (3.94%) Elasticsearch
14 51 (3.94%) Hadoop
15 45 (3.48%) Azure SQL Database
16 44 (3.40%) Azure SQL Data Warehouse
17 39 (3.02%) Star Schema
18 36 (2.78%) Relational Database
19 34 (2.63%) Delta Lake
Development Applications
1 284 (21.96%) Git
2 109 (8.43%) Jenkins
3 71 (5.49%) Bitbucket
4 65 (5.03%) Artifactory
5 50 (3.87%) GitLab
6 16 (1.24%) Spinnaker
7 9 (0.70%) CircleCI
8 8 (0.62%) JIRA
8 8 (0.62%) Oracle Forms
9 6 (0.46%) RStudio
10 5 (0.39%) GDB
11 4 (0.31%) Octopus Deploy
12 3 (0.23%) Visual Studio
13 2 (0.15%) MLflow
13 2 (0.15%) TeamCity
13 2 (0.15%) Visual Studio Team System
13 2 (0.15%) webpack
14 1 (0.077%) Cucumber
14 1 (0.077%) Mockito
14 1 (0.077%) SpecFlow
General
1 448 (34.65%) Analytical Skills
2 357 (27.61%) Social Skills
3 302 (23.36%) Finance
4 203 (15.70%) Inclusion and Diversity
5 173 (13.38%) Presentation Skills
6 170 (13.15%) Law
7 166 (12.84%) Retail
8 121 (9.36%) Marketing
9 75 (5.80%) Banking
10 51 (3.94%) Legal
11 37 (2.86%) Public Sector
12 21 (1.62%) Games
13 20 (1.55%) Investment Banking
14 19 (1.47%) Financial Institution
15 16 (1.24%) Billing
16 14 (1.08%) Front Office
17 12 (0.93%) Back Office
18 11 (0.85%) Influencing Skills
18 11 (0.85%) Telecoms
19 7 (0.54%) Health Technology
Job Titles
1 669 (51.74%) Data Engineer
2 365 (28.23%) Senior
3 180 (13.92%) Principal Engineer
4 178 (13.77%) Platform Engineer
5 177 (13.69%) Principal Data Engineer
6 174 (13.46%) Lead
7 162 (12.53%) Analyst
8 153 (11.83%) Data Analyst
9 135 (10.44%) Software Engineer
10 128 (9.90%) Senior Data Engineer
11 105 (8.12%) Senior Analyst
12 103 (7.97%) Senior Data Analyst
13 84 (6.50%) Developer
14 69 (5.34%) Lead Software Engineer
15 65 (5.03%) Architect
16 59 (4.56%) Data Architect
17 58 (4.49%) Senior Software Engineer
18 54 (4.18%) AWS Engineer
19 50 (3.87%) Lead Data Engineer
20 48 (3.71%) Azure Engineer
Libraries, Frameworks & Software Standards
1 94 (7.27%) Kafka
2 78 (6.03%) XML
3 77 (5.96%) PySpark
4 75 (5.80%) JSON
5 73 (5.65%) RESTful
6 71 (5.49%) Web Services
7 65 (5.03%) JSON-LD
7 65 (5.03%) LDAP
7 65 (5.03%) OpenID
7 65 (5.03%) SAML
8 49 (3.79%) Pandas
9 34 (2.63%) Apache Beam
10 29 (2.24%) React
11 22 (1.70%) Apache NiFi
12 21 (1.62%) PyTorch
12 21 (1.62%) TensorFlow
13 20 (1.55%) .NET
13 20 (1.55%) XGBoost
14 17 (1.31%) REST
15 15 (1.16%) .NET Core
Miscellaneous
1 213 (16.47%) Self-Motivation
2 102 (7.89%) Data Structures
3 101 (7.81%) Management Information System
4 83 (6.42%) Algorithms
5 74 (5.72%) Cloud Native
6 65 (5.03%) Enterprise Cloud
6 65 (5.03%) Onboarding
6 65 (5.03%) Reference Data
7 46 (3.56%) Product Ownership
8 25 (1.93%) Public Cloud
9 23 (1.78%) Taxonomies
10 20 (1.55%) IoT
11 19 (1.47%) Hedge funds
12 14 (1.08%) Greenfield Project
13 12 (0.93%) Distributed Systems
13 12 (0.93%) Enterprise Software
14 11 (0.85%) Blog
14 11 (0.85%) Private Cloud
14 11 (0.85%) Reinsurance
15 9 (0.70%) Client/Server
Operating Systems
1 42 (3.25%) Linux
2 21 (1.62%) Unix
3 2 (0.15%) Windows
4 1 (0.077%) Android
4 1 (0.077%) Apple iOS
4 1 (0.077%) Windows Server
Processes & Methodologies
1 540 (41.76%) Agile
2 517 (39.98%) Data Engineering
3 510 (39.44%) Analytics
4 495 (38.28%) Problem-Solving
5 452 (34.96%) ETL
6 424 (32.79%) Data Science
7 403 (31.17%) Data Modelling
8 368 (28.46%) Visualisation
9 333 (25.75%) CI/CD
10 310 (23.98%) Data Analysis
11 281 (21.73%) DevOps
12 272 (21.04%) Machine Learning
13 260 (20.11%) Mentoring
14 253 (19.57%) Business Intelligence
15 237 (18.33%) Software Engineering
16 219 (16.94%) Platform Engineering
17 189 (14.62%) Artificial Intelligence
17 189 (14.62%) DataOps
18 178 (13.77%) Mathematics
19 168 (12.99%) Statistics
Programming Languages
1 810 (62.65%) SQL
2 745 (57.62%) Python
3 171 (13.23%) Java
4 100 (7.73%) Scala
5 84 (6.50%) Go
6 65 (5.03%) Prolog
6 65 (5.03%) SPARQL
7 55 (4.25%) T-SQL
8 38 (2.94%) C#
9 31 (2.40%) R
10 29 (2.24%) JavaScript
11 26 (2.01%) TypeScript
12 20 (1.55%) PL/SQL
13 19 (1.47%) Kotlin
14 17 (1.31%) Data Analysis Expressions
15 9 (0.70%) C
16 8 (0.62%) Clojure
17 7 (0.54%) MATLAB
18 6 (0.46%) PowerShell
19 3 (0.23%) Rust
Qualifications
1 288 (22.27%) Degree
2 174 (13.46%) DBS Check
3 76 (5.88%) Computer Science Degree
4 48 (3.71%) Master's Degree
5 33 (2.55%) Security Cleared
6 25 (1.93%) SC Cleared
7 15 (1.16%) DV Cleared
7 15 (1.16%) Microsoft Certification
8 12 (0.93%) Azure Certification
9 10 (0.77%) PhD
10 5 (0.39%) Numerate Degree
11 4 (0.31%) CIMA
11 4 (0.31%) PMI Certification
12 3 (0.23%) AWS Certification
12 3 (0.23%) AWS Certified Solutions Architect
12 3 (0.23%) BPSS Clearance
12 3 (0.23%) SnowPro
13 1 (0.077%) Postgraduate
Quality Assurance & Compliance
1 183 (14.15%) GDPR
2 181 (14.00%) Data Quality
3 22 (1.70%) QA
4 18 (1.39%) SLA
5 12 (0.93%) Accessibility
6 4 (0.31%) Solvency II
7 3 (0.23%) ISO/IEC 27001
8 1 (0.077%) California Consumer Privacy Act
8 1 (0.077%) HIPAA
8 1 (0.077%) NCSC
System Software
1 67 (5.18%) Active Directory
2 51 (3.94%) Docker
3 9 (0.70%) HDFS
4 2 (0.15%) Virtual Machines
Systems Management
1 102 (7.89%) Kubernetes
2 55 (4.25%) Terraform
3 14 (1.08%) Ansible
4 7 (0.54%) Argo
5 4 (0.31%) Kubeflow
6 1 (0.077%) Grafana
6 1 (0.077%) Prometheus
Vendors
1 390 (30.16%) Databricks
2 230 (17.79%) Google
3 145 (11.21%) Microsoft
4 50 (3.87%) Oracle
5 29 (2.24%) Alteryx
6 20 (1.55%) Informatica
7 17 (1.31%) Cognos
7 17 (1.31%) Sybase
8 12 (0.93%) Cloudera
8 12 (0.93%) OpenAI
9 10 (0.77%) SAP
9 10 (0.77%) Teradata
9 10 (0.77%) YouTube
10 7 (0.54%) Bloomberg
11 6 (0.46%) Dataiku
11 6 (0.46%) F5
11 6 (0.46%) Talend
12 5 (0.39%) Confluent
13 4 (0.31%) Hortonworks
14 3 (0.23%) SnapLogic