Data Pipeline Jobs

Data Pipelines
UK

The following table provides summary statistics for permanent job vacancies with a requirement for Data Pipeline skills. Included is a benchmarking guide to the salaries offered in vacancies that have cited Data Pipeline over the 6 months to 8 June 2024 with a comparison to the same period in the previous 2 years.

6 months to
8 Jun 2024
Same period 2023 Same period 2022
Rank 226 282 218
Rank change year-on-year +56 -64 +73
Permanent jobs citing Data Pipeline 1,299 1,197 3,335
As % of all permanent jobs advertised in the UK 1.20% 1.28% 1.94%
As % of the Processes & Methodologies category 1.45% 1.34% 2.02%
Number of salaries quoted 700 696 1,071
10th Percentile £37,000 £46,625 £46,250
25th Percentile £50,000 £56,250 £57,500
Median annual salary (50th Percentile) £67,500 £69,447 £75,000
Median % change year-on-year -2.80% -7.40% +11.11%
75th Percentile £85,000 £87,419 £87,500
90th Percentile £105,000 £100,000 £100,000
UK excluding London median annual salary £60,000 £61,000 £65,000
% change year-on-year -1.64% -6.15% +12.07%

All Process and Methodology Skills
UK

Data Pipeline is in the Processes and Methodologies category. The following table is for comparison with the above and provides summary statistics for all permanent job vacancies with a requirement for process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 89,730 89,109 165,036
As % of all permanent jobs advertised in the UK 82.98% 95.44% 95.91%
Number of salaries quoted 62,034 53,947 83,947
10th Percentile £29,500 £34,000 £33,500
25th Percentile £40,000 £45,000 £44,000
Median annual salary (50th Percentile) £55,000 £60,800 £60,000
Median % change year-on-year -9.54% +1.33% +9.09%
75th Percentile £72,500 £81,250 £80,000
90th Percentile £92,500 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +9.38%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 8 June 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 8 June 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +38 1,076 £67,500 -3.57% 307
UK excluding London +43 746 £60,000 -1.64% 148
Work from Home +10 539 £67,500 -1.92% 157
London +69 502 £75,000 -6.25% 175
North of England +11 233 £52,500 -19.23% 55
Scotland +29 173 £66,500 +2.31% 17
North West +48 146 £50,000 -10.80% 33
South East +31 112 £67,500 +15.40% 23
Midlands -5 111 £52,000 -9.57% 27
South West -70 73 £67,500 -1.69% 12
West Midlands +1 59 £60,000 +2.13% 19
East Midlands -1 52 £40,000 -27.27% 6
Yorkshire -24 47 £70,000 +0.80% 17
North East +21 40 £52,500 +34.62% 5
East of England +5 29 £57,500 -9.80% 8
Wales +9 16 £46,131 +6.05% 2
Northern Ireland +10 3 £60,000 -20.00% 5

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 150 (11.55%) Apache Spark
2 95 (7.31%) Apache
3 73 (5.62%) Apache Airflow
4 11 (0.85%) SAS
5 6 (0.46%) Confluence
6 5 (0.38%) Luigi
6 5 (0.38%) SharePoint
7 3 (0.23%) Microsoft Exchange
8 1 (0.077%) Apache Pulsar
Applications
1 23 (1.77%) Microsoft Excel
2 9 (0.69%) Microsoft Office
3 4 (0.31%) Spreadsheet
4 3 (0.23%) Microsoft PowerPoint
4 3 (0.23%) MS Visio
5 1 (0.077%) Revit
Business Applications
1 4 (0.31%) Primavera
2 1 (0.077%) Salesforce CRM
2 1 (0.077%) SAP S/4HANA
2 1 (0.077%) Shopify
Cloud Services
1 511 (39.34%) AWS
2 469 (36.10%) Azure
3 355 (27.33%) GCP
4 202 (15.55%) Power Platform
5 164 (12.63%) Azure Data Factory
6 152 (11.70%) Snowflake
7 83 (6.39%) Serverless
8 76 (5.85%) Azure Synapse Analytics
9 63 (4.85%) Amazon S3
9 63 (4.85%) dbt
10 57 (4.39%) Cloud Computing
11 56 (4.31%) Matillion
12 42 (3.23%) Azure DevOps
13 39 (3.00%) AWS Lambda
14 37 (2.85%) GitHub
15 36 (2.77%) Azure Cosmos DB
16 28 (2.16%) Amazon EMR
17 22 (1.69%) SaaS
18 20 (1.54%) AWS Step Functions
18 20 (1.54%) Fivetran
Communications & Networking
1 65 (5.00%) Kerberos
2 13 (1.00%) DSL
3 4 (0.31%) Firewall
4 2 (0.15%) Intrusion Detection
4 2 (0.15%) Network Security
4 2 (0.15%) VPN
5 1 (0.077%) 3GPP
5 1 (0.077%) 5G
5 1 (0.077%) PPP
Database & Business Intelligence
1 358 (27.56%) Data Warehouse
2 241 (18.55%) Data Lake
3 228 (17.55%) BigQuery
4 214 (16.47%) Tableau
5 198 (15.24%) Power BI
6 187 (14.40%) Looker
7 145 (11.16%) SQL Server
8 137 (10.55%) Big Data
9 123 (9.47%) Metadata
9 123 (9.47%) NoSQL
10 76 (5.85%) SQL Server Integration Services
11 66 (5.08%) Amazon Redshift
12 56 (4.31%) PostgreSQL
13 52 (4.00%) Elasticsearch
14 51 (3.93%) Hadoop
15 50 (3.85%) Azure SQL Database
16 43 (3.31%) Azure SQL Data Warehouse
17 40 (3.08%) Star Schema
18 37 (2.85%) Relational Database
19 32 (2.46%) Delta Lake
Development Applications
1 285 (21.94%) Git
2 104 (8.01%) Jenkins
3 70 (5.39%) Bitbucket
4 65 (5.00%) Artifactory
5 49 (3.77%) GitLab
6 14 (1.08%) Spinnaker
7 11 (0.85%) JIRA
8 8 (0.62%) CircleCI
8 8 (0.62%) Oracle Forms
9 6 (0.46%) RStudio
10 5 (0.38%) GDB
11 4 (0.31%) Octopus Deploy
11 4 (0.31%) Visual Studio
12 2 (0.15%) MLflow
12 2 (0.15%) webpack
13 1 (0.077%) ERwin
13 1 (0.077%) Mockito
13 1 (0.077%) SpecFlow
13 1 (0.077%) TeamCity
13 1 (0.077%) Visual Studio Team System
General
1 448 (34.49%) Analytical Skills
2 363 (27.94%) Social Skills
3 296 (22.79%) Finance
4 190 (14.63%) Inclusion and Diversity
5 169 (13.01%) Presentation Skills
6 167 (12.86%) Retail
7 158 (12.16%) Law
8 112 (8.62%) Marketing
9 68 (5.23%) Banking
10 47 (3.62%) Legal
11 39 (3.00%) Public Sector
12 21 (1.62%) Games
13 16 (1.23%) Billing
13 16 (1.23%) Investment Banking
14 15 (1.15%) Financial Institution
15 14 (1.08%) Front Office
16 12 (0.92%) Back Office
17 11 (0.85%) Influencing Skills
17 11 (0.85%) Telecoms
18 7 (0.54%) Health Technology
Job Titles
1 672 (51.73%) Data Engineer
2 372 (28.64%) Senior
3 181 (13.93%) Principal Engineer
4 179 (13.78%) Principal Data Engineer
5 177 (13.63%) Lead
5 177 (13.63%) Platform Engineer
6 160 (12.32%) Analyst
7 153 (11.78%) Data Analyst
8 134 (10.32%) Senior Data Engineer
9 131 (10.08%) Software Engineer
10 104 (8.01%) Senior Analyst
11 103 (7.93%) Senior Data Analyst
12 86 (6.62%) Developer
13 69 (5.31%) Architect
13 69 (5.31%) Lead Software Engineer
14 63 (4.85%) Data Architect
15 59 (4.54%) Senior Software Engineer
16 53 (4.08%) AWS Engineer
16 53 (4.08%) Lead Data Engineer
17 47 (3.62%) Azure Engineer
Libraries, Frameworks & Software Standards
1 87 (6.70%) Kafka
2 83 (6.39%) PySpark
3 77 (5.93%) XML
4 75 (5.77%) JSON
5 72 (5.54%) RESTful
6 71 (5.47%) Web Services
7 65 (5.00%) JSON-LD
7 65 (5.00%) LDAP
7 65 (5.00%) OpenID
7 65 (5.00%) SAML
8 50 (3.85%) Pandas
9 33 (2.54%) Apache Beam
10 31 (2.39%) React
11 21 (1.62%) PyTorch
11 21 (1.62%) TensorFlow
12 20 (1.54%) .NET
13 19 (1.46%) XGBoost
14 18 (1.39%) Apache NiFi
15 17 (1.31%) REST
16 15 (1.15%) .NET Core
Miscellaneous
1 214 (16.47%) Self-Motivation
2 106 (8.16%) Data Structures
3 97 (7.47%) Management Information System
4 82 (6.31%) Algorithms
5 65 (5.00%) Cloud Native
5 65 (5.00%) Enterprise Cloud
5 65 (5.00%) Reference Data
6 61 (4.70%) Onboarding
7 42 (3.23%) Product Ownership
8 21 (1.62%) Public Cloud
9 20 (1.54%) Hedge funds
9 20 (1.54%) IoT
10 19 (1.46%) Taxonomies
11 15 (1.15%) Greenfield Project
12 12 (0.92%) Reinsurance
13 11 (0.85%) Blog
13 11 (0.85%) Distributed Systems
13 11 (0.85%) Enterprise Software
13 11 (0.85%) Private Cloud
14 9 (0.69%) Client/Server
Operating Systems
1 40 (3.08%) Linux
2 20 (1.54%) Unix
3 2 (0.15%) Android
3 2 (0.15%) Windows
4 1 (0.077%) Apple iOS
4 1 (0.077%) Windows Server
Processes & Methodologies
1 537 (41.34%) Agile
2 528 (40.65%) Data Engineering
3 518 (39.88%) Analytics
4 497 (38.26%) Problem-Solving
5 454 (34.95%) ETL
6 423 (32.56%) Data Science
7 399 (30.72%) Data Modelling
8 362 (27.87%) Visualisation
9 326 (25.10%) CI/CD
10 300 (23.09%) Data Analysis
11 281 (21.63%) DevOps
12 264 (20.32%) Mentoring
13 259 (19.94%) Machine Learning
14 250 (19.25%) Business Intelligence
15 226 (17.40%) Software Engineering
16 218 (16.78%) Platform Engineering
17 189 (14.55%) DataOps
18 186 (14.32%) Artificial Intelligence
19 180 (13.86%) Mathematics
20 168 (12.93%) Statistics
Programming Languages
1 813 (62.59%) SQL
2 741 (57.04%) Python
3 165 (12.70%) Java
4 98 (7.54%) Scala
5 84 (6.47%) Go
6 65 (5.00%) Prolog
6 65 (5.00%) SPARQL
7 55 (4.23%) T-SQL
8 37 (2.85%) C#
9 31 (2.39%) R
10 29 (2.23%) JavaScript
10 29 (2.23%) TypeScript
11 20 (1.54%) Kotlin
12 17 (1.31%) Data Analysis Expressions
13 16 (1.23%) PL/SQL
14 9 (0.69%) C
14 9 (0.69%) Clojure
15 7 (0.54%) MATLAB
16 6 (0.46%) PowerShell
17 4 (0.31%) C++
Qualifications
1 290 (22.32%) Degree
2 176 (13.55%) DBS Check
3 76 (5.85%) Computer Science Degree
4 48 (3.70%) Master's Degree
5 38 (2.93%) Security Cleared
6 30 (2.31%) SC Cleared
7 19 (1.46%) DV Cleared
8 15 (1.15%) Microsoft Certification
9 12 (0.92%) Azure Certification
10 10 (0.77%) PhD
11 5 (0.38%) Numerate Degree
12 4 (0.31%) CIMA
12 4 (0.31%) PMI Certification
13 3 (0.23%) AWS Certification
13 3 (0.23%) AWS Certified Solutions Architect
13 3 (0.23%) BPSS Clearance
14 2 (0.15%) SnowPro
15 1 (0.077%) Postgraduate
Quality Assurance & Compliance
1 189 (14.55%) Data Quality
2 183 (14.09%) GDPR
3 22 (1.69%) QA
4 14 (1.08%) SLA
5 12 (0.92%) Accessibility
6 4 (0.31%) Solvency II
7 3 (0.23%) ISO/IEC 27001
8 1 (0.077%) California Consumer Privacy Act
8 1 (0.077%) HIPAA
8 1 (0.077%) NCSC
System Software
1 69 (5.31%) Active Directory
2 49 (3.77%) Docker
3 8 (0.62%) HDFS
4 4 (0.31%) Virtual Machines
Systems Management
1 95 (7.31%) Kubernetes
2 53 (4.08%) Terraform
3 14 (1.08%) Ansible
4 7 (0.54%) Argo
5 4 (0.31%) Kubeflow
6 1 (0.077%) Grafana
6 1 (0.077%) Prometheus
Vendors
1 381 (29.33%) Databricks
2 233 (17.94%) Google
3 148 (11.39%) Microsoft
4 44 (3.39%) Oracle
5 25 (1.92%) Alteryx
6 22 (1.69%) Informatica
7 13 (1.00%) Cognos
7 13 (1.00%) Sybase
8 12 (0.92%) OpenAI
9 11 (0.85%) Cloudera
10 10 (0.77%) SAP
10 10 (0.77%) Teradata
10 10 (0.77%) YouTube
11 7 (0.54%) Bloomberg
12 6 (0.46%) Dataiku
12 6 (0.46%) F5
12 6 (0.46%) Talend
13 4 (0.31%) Hortonworks
14 3 (0.23%) Salesforce
14 3 (0.23%) Splunk