Data Pipeline Jobs

Data Pipelines
UK

The table below provides summary statistics for permanent job vacancies requiring Data Pipeline skills. It includes a benchmarking guide to the annual salaries offered in vacancies that cited Data Pipeline over the 6 months leading up to 26 July 2024, comparing them to the same period in the previous two years.

6 months to
26 Jul 2024
Same period 2023 Same period 2022
Rank 241 285 212
Rank change year-on-year +44 -73 +63
Permanent jobs citing Data Pipeline 1,023 1,004 3,447
As % of all permanent jobs advertised in the UK 1.13% 1.24% 1.97%
As % of the Processes & Methodologies category 1.38% 1.31% 2.04%
Number of salaries quoted 488 672 1,180
10th Percentile £46,250 £47,500 £47,500
25th Percentile £56,250 £56,250 £58,750
Median annual salary (50th Percentile) £70,000 £70,000 £75,000
Median % change year-on-year - -6.67% +11.11%
75th Percentile £88,750 £86,250 £87,500
90th Percentile £103,825 £97,500 £100,000
UK excluding London median annual salary £65,599 £65,000 £70,000
% change year-on-year +0.92% -7.14% +21.74%

All Process and Methodology Skills
UK

Data Pipeline falls under the Processes and Methodologies category. For comparison with the information above, the following table provides summary statistics for all permanent job vacancies requiring process or methodology skills.

Permanent vacancies with a requirement for process or methodology skills 73,969 76,929 168,958
As % of all permanent jobs advertised in the UK 81.84% 95.34% 96.40%
Number of salaries quoted 48,628 50,249 85,709
10th Percentile £30,000 £33,750 £33,750
25th Percentile £40,000 £45,000 £44,250
Median annual salary (50th Percentile) £55,000 £60,000 £60,000
Median % change year-on-year -8.33% - +9.09%
75th Percentile £73,750 £80,000 £80,000
90th Percentile £93,750 £100,000 £96,250
UK excluding London median annual salary £50,000 £55,000 £52,500
% change year-on-year -9.09% +4.76% +6.06%

Data Pipeline
Job Vacancy Trend

Job postings citing Data Pipeline as a proportion of all IT jobs advertised.

Job vacancy trend for Data Pipeline in the UK

Data Pipeline
Salary Trend

3-month moving average salary quoted in jobs citing Data Pipeline.

Salary trend for Data Pipeline in the UK

Data Pipeline
Salary Histogram

Salary distribution for jobs citing Data Pipeline over the 6 months to 26 July 2024.

Salary histogram for Data Pipeline in the UK

Data Pipeline
Top 17 Job Locations

The table below looks at the demand and provides a guide to the median salaries quoted in IT jobs citing Data Pipeline within the UK over the 6 months to 26 July 2024. The 'Rank Change' column provides an indication of the change in demand within each location based on the same 6 month period last year.

Location Rank Change
on Same Period
Last Year
Matching
Permanent
IT Job Ads
Median Salary
Past 6 Months
Median Salary
% Change
on Same Period
Last Year
Live
Jobs
England +34 881 £70,000 - 88
UK excluding London +22 552 £65,599 +0.92% 78
London +66 431 £75,000 - 31
Work from Home +24 408 £70,000 +1.71% 66
North of England +32 166 £60,000 -7.69% 14
North West +25 107 £57,000 +6.05% 9
Scotland +4 105 £67,500 -2.80% 6
South East +31 93 £67,500 +3.85% 18
Midlands +4 93 £62,500 +9.17% 15
South West -49 61 £62,500 -10.00% 12
East Midlands -1 50 £50,750 -7.73% 5
West Midlands -3 43 £69,617 +21.07% 10
Yorkshire +32 38 £70,000 +0.80% 2
East of England +4 23 £57,500 - 3
North East +6 21 £75,000 +51.52% 3
Wales +23 14 £45,000 +0.65% 1
Channel Islands - 1 - - 1

Data Pipeline
Co-occurring Skills and Capabilities by Category

The follow tables expand on the table above by listing co-occurrences grouped by category. The same employment type, locality and period is covered with up to 20 co-occurrences shown in each of the following categories:

Application Platforms
1 100 (9.78%) Apache Spark
2 60 (5.87%) Apache
3 56 (5.47%) Apache Airflow
4 12 (1.17%) Confluence
5 10 (0.98%) SAS
6 4 (0.39%) SharePoint
7 3 (0.29%) Luigi
8 2 (0.20%) Microsoft Exchange
9 1 (0.098%) Apache Pulsar
Applications
1 20 (1.96%) Microsoft Excel
2 9 (0.88%) Microsoft Office
3 2 (0.20%) Microsoft PowerPoint
3 2 (0.20%) MS Visio
4 1 (0.098%) Revit
Business Applications
1 2 (0.20%) Primavera
1 2 (0.20%) Salesforce CRM
1 2 (0.20%) SAP S/4HANA
2 1 (0.098%) Shopify
Cloud Services
1 425 (41.54%) AWS
2 345 (33.72%) Azure
3 321 (31.38%) GCP
4 148 (14.47%) Power Platform
5 138 (13.49%) Azure Data Factory
6 79 (7.72%) Azure Synapse Analytics
7 73 (7.14%) Snowflake
8 62 (6.06%) Serverless
9 50 (4.89%) Azure DevOps
10 49 (4.79%) Amazon S3
11 47 (4.59%) Cloud Computing
12 44 (4.30%) dbt
13 39 (3.81%) AWS Lambda
14 33 (3.23%) GitHub
15 24 (2.35%) Amazon EMR
16 19 (1.86%) AWS Step Functions
16 19 (1.86%) SaaS
17 17 (1.66%) PowerApps
18 14 (1.37%) Google Gemini
19 13 (1.27%) Power Automate
Communications & Networking
1 44 (4.30%) Kerberos
2 2 (0.20%) Intrusion Detection
2 2 (0.20%) VPN
3 1 (0.098%) 3GPP
3 1 (0.098%) 5G
3 1 (0.098%) Firewall
Database & Business Intelligence
1 237 (23.17%) Data Warehouse
2 204 (19.94%) BigQuery
3 197 (19.26%) Tableau
4 185 (18.08%) Looker
5 156 (15.25%) Data Lake
6 144 (14.08%) Power BI
7 102 (9.97%) NoSQL
8 79 (7.72%) Big Data
8 79 (7.72%) Metadata
9 71 (6.94%) SQL Server
10 61 (5.96%) Azure SQL Database
11 42 (4.11%) PostgreSQL
12 41 (4.01%) Amazon Redshift
13 38 (3.71%) SQL Server Integration Services
14 26 (2.54%) Hadoop
15 25 (2.44%) Relational Database
16 23 (2.25%) SQL Server Reporting Services
17 22 (2.15%) Amazon Athena
18 21 (2.05%) Data Mining
19 19 (1.86%) MySQL
Development Applications
1 256 (25.02%) Git
2 70 (6.84%) Jenkins
3 48 (4.69%) Bitbucket
4 44 (4.30%) Artifactory
5 21 (2.05%) GitLab
6 18 (1.76%) JIRA
7 10 (0.98%) Spinnaker
8 6 (0.59%) RStudio
9 5 (0.49%) GDB
9 5 (0.49%) MLflow
9 5 (0.49%) Visual Studio
10 4 (0.39%) CircleCI
10 4 (0.39%) Octopus Deploy
11 2 (0.20%) Flyway
11 2 (0.20%) Liquibase
11 2 (0.20%) Oracle Forms
12 1 (0.098%) ERwin
12 1 (0.098%) Mockito
12 1 (0.098%) SpecFlow
General
1 398 (38.91%) Analytical Skills
2 327 (31.96%) Social Skills
3 189 (18.48%) Finance
4 166 (16.23%) Retail
5 154 (15.05%) Presentation Skills
6 113 (11.05%) Inclusion and Diversity
7 97 (9.48%) Law
8 54 (5.28%) Marketing
9 37 (3.62%) Legal
10 23 (2.25%) Banking
11 16 (1.56%) Public Sector
12 14 (1.37%) Front Office
13 10 (0.98%) Billing
13 10 (0.98%) Telecoms
14 9 (0.88%) Back Office
14 9 (0.88%) Games
15 7 (0.68%) Advertising
15 7 (0.68%) Influencing Skills
16 4 (0.39%) Documentation Skills
16 4 (0.39%) Publishing
Job Titles
1 517 (50.54%) Data Engineer
2 309 (30.21%) Senior
3 180 (17.60%) Principal Engineer
4 179 (17.50%) Principal Data Engineer
5 170 (16.62%) Platform Engineer
6 159 (15.54%) Analyst
7 155 (15.15%) Lead
8 154 (15.05%) Data Analyst
9 110 (10.75%) Senior Analyst
10 109 (10.65%) Senior Data Analyst
10 109 (10.65%) Software Engineer
11 101 (9.87%) Senior Data Engineer
12 61 (5.96%) Lead Software Engineer
13 53 (5.18%) Developer
14 50 (4.89%) Azure Engineer
15 49 (4.79%) Senior Software Engineer
16 48 (4.69%) AWS Engineer
17 47 (4.59%) Architect
18 44 (4.30%) Azure Data Engineer
19 40 (3.91%) Lead Data Engineer
Libraries, Frameworks & Software Standards
1 69 (6.74%) PySpark
2 60 (5.87%) Kafka
3 54 (5.28%) XML
4 50 (4.89%) JSON
4 50 (4.89%) Web Services
5 47 (4.59%) RESTful
6 44 (4.30%) JSON-LD
6 44 (4.30%) LDAP
6 44 (4.30%) OpenID
6 44 (4.30%) SAML
7 35 (3.42%) React
8 29 (2.83%) Pandas
9 22 (2.15%) PyTorch
9 22 (2.15%) TensorFlow
10 19 (1.86%) Apache Beam
11 16 (1.56%) .NET
12 14 (1.37%) REST
13 12 (1.17%) .NET Core
14 11 (1.08%) .NET Framework
14 11 (1.08%) Unity DI Container
Miscellaneous
1 192 (18.77%) Self-Motivation
2 67 (6.55%) Algorithms
3 62 (6.06%) Data Structures
4 52 (5.08%) Management Information System
5 48 (4.69%) Onboarding
6 44 (4.30%) Enterprise Cloud
6 44 (4.30%) Reference Data
7 26 (2.54%) Distributed Systems
8 24 (2.35%) Cloud Native
9 20 (1.96%) Hedge funds
10 15 (1.47%) Product Ownership
11 13 (1.27%) Media Streaming
12 12 (1.17%) Blog
13 11 (1.08%) Private Cloud
14 9 (0.88%) IoT
15 8 (0.78%) Enterprise Software
15 8 (0.78%) Legacy Applications
15 8 (0.78%) Reinsurance
15 8 (0.78%) Renewable Energy
16 7 (0.68%) Greenfield Project
Operating Systems
1 33 (3.23%) Linux
2 18 (1.76%) Unix
3 2 (0.20%) Windows
4 1 (0.098%) Android
4 1 (0.098%) Windows Server
Processes & Methodologies
1 457 (44.67%) Problem-Solving
2 435 (42.52%) Agile
3 422 (41.25%) Analytics
4 417 (40.76%) Data Engineering
5 323 (31.57%) Data Science
6 304 (29.72%) Visualisation
7 286 (27.96%) ETL
8 283 (27.66%) DevOps
9 256 (25.02%) Data Modelling
10 245 (23.95%) Data Analysis
11 232 (22.68%) Mentoring
12 221 (21.60%) CI/CD
13 209 (20.43%) Platform Engineering
14 181 (17.69%) DataOps
14 181 (17.69%) Mathematics
15 179 (17.50%) Software Engineering
16 169 (16.52%) Change Management
17 168 (16.42%) Statistics
18 164 (16.03%) Actionable Insight
19 161 (15.74%) Machine Learning
Programming Languages
1 595 (58.16%) SQL
2 555 (54.25%) Python
3 135 (13.20%) Java
4 71 (6.94%) Scala
5 54 (5.28%) Go
6 44 (4.30%) Prolog
6 44 (4.30%) SPARQL
7 33 (3.23%) Kotlin
8 32 (3.13%) JavaScript
9 30 (2.93%) T-SQL
10 28 (2.74%) TypeScript
11 26 (2.54%) C#
11 26 (2.54%) R
12 17 (1.66%) PowerShell
13 10 (0.98%) Clojure
14 7 (0.68%) Bash
14 7 (0.68%) Data Analysis Expressions
14 7 (0.68%) MATLAB
14 7 (0.68%) Rust
14 7 (0.68%) Shell Script
Qualifications
1 274 (26.78%) Degree
2 170 (16.62%) DBS Check
3 65 (6.35%) Computer Science Degree
4 46 (4.50%) Master's Degree
5 38 (3.71%) Security Cleared
6 31 (3.03%) SC Cleared
7 20 (1.96%) DV Cleared
8 16 (1.56%) Azure Certification
9 11 (1.08%) PhD
10 7 (0.68%) Microsoft Certification
11 3 (0.29%) AWS Certification
11 3 (0.29%) AWS Certified Solutions Architect
12 2 (0.20%) CIMA
12 2 (0.20%) Numerate Degree
12 2 (0.20%) PMI Certification
13 1 (0.098%) BPSS Clearance
13 1 (0.098%) Postgraduate
13 1 (0.098%) SnowPro
Quality Assurance & Compliance
1 169 (16.52%) GDPR
2 168 (16.42%) Data Quality
3 21 (2.05%) QA
4 8 (0.78%) Accessibility
5 3 (0.29%) ISO/IEC 27001
6 2 (0.20%) California Consumer Privacy Act
6 2 (0.20%) HIPAA
6 2 (0.20%) ISO 20022
7 1 (0.098%) NCSC
System Software
1 50 (4.89%) Active Directory
2 41 (4.01%) Docker
3 18 (1.76%) Virtual Machines
4 4 (0.39%) HDFS
Systems Management
1 68 (6.65%) Kubernetes
2 59 (5.77%) Terraform
3 15 (1.47%) Ansible
4 6 (0.59%) Argo
5 3 (0.29%) Kubeflow
6 1 (0.098%) Grafana
6 1 (0.098%) Prometheus
Vendors
1 305 (29.81%) Databricks
2 237 (23.17%) Google
3 128 (12.51%) Microsoft
4 18 (1.76%) Oracle
5 15 (1.47%) Informatica
6 12 (1.17%) OpenAI
7 10 (0.98%) Alteryx
8 8 (0.78%) SAP
9 7 (0.68%) Bloomberg
10 6 (0.59%) Cloudera
11 5 (0.49%) Splunk
12 4 (0.39%) Foundry
12 4 (0.39%) Palantir
12 4 (0.39%) Sage
12 4 (0.39%) Salesforce
12 4 (0.39%) Talend
13 3 (0.29%) Ab Initio
13 3 (0.29%) Hortonworks
13 3 (0.29%) SnapLogic
14 2 (0.20%) YouTube