101 to 125 of 132 PySpark Jobs in England

work independently and collaborate within cross-functional teams Some knowledge of programming languages such as Python (preferred) Experience with data processing technologies such as PySpark (preferred) Familiarity with AWS services and cloud-based data workflows (preferred ...

data freshness. Any AI/ML knowledge specifically around LLM's/RAG would also be useful. Data Engineer Key Skills: Python/Pyspark Databricks Delta Lake Azure LLM's/RAG Location: Remote Rate: Circa £450/Day (Outside) Contract Length: 6-12 Months Initial If this Data ...

What We're Looking For Strong data modelling expertise and experience with Data Lineage Maps. Hands-on skills with SQL/Python, Synapse Notebooks, PySpark, and Delta Lake. Familiarity with MS Fabric and medallion architecture. Excellent communication and stakeholder management skills. Agile mindset and ability to thrive in cross ...

What We're Looking For Strong data modelling expertise and experience with Data Lineage Maps. Hands-on skills with SQL/Python, Synapse Notebooks, PySpark, and Delta Lake. Familiarity with MS Fabric and medallion architecture. Excellent communication and stakeholder management skills. Agile mindset and ability to thrive in cross ...

complex operational challenges. You'll design and implement scalable generative AI workflows, often using technologies like Palantir AIP, while building robust data pipelines with PySpark, Python, and SQL. A key responsibility is executing sophisticated data integrations across distributed systems and enterprise environments, including ERPs and CRMs. You'll collaborate ...

Defining and enforcing engineering standards across code quality, testing, CI/CD, IaC, and observability Building and optimising ELT/ETL pipelines using Glue (PySpark) , Lambda, Step Functions, and event-driven patterns Championing security-by-design in collaboration with security and assurance teams (IAM, KMS, encryption, auditability) Implementing … across data analytics and data engineering domains Strong hands-on experience with: Amazon S3 (data lake patterns, partitioning, lifecycle management) AWS Glue (Jobs, Crawlers, PySpark, orchestration) Amazon Redshift (data modelling, performance tuning, WLM, Spectrum) Advanced Python (PySpark) and SQL skills Experience implementing CI/CD pipelines and Infrastructure ...

expertise in data processing within a cloud-based data platform. Junior Data Engineer duties: Develop, test, and maintain scalable data pipelines using Python and PySpark within an Azure Databricks development environment Clean, aggregate, and transform complex datasets to meet business requirements. Assist with data quality checks and support data … Junior Data Engineer, you will have: Strong proficiency in Python, particularly for data manipulation and automation scripts; experience in libraries such as pandas, PySpark, numpy, pyodbc or associated would be beneficial Hands-on experience (or academic project experience) using PySpark. An understanding of ETL/ELT processes and/ ...

balance What you will be doing Contributing to the design and delivery of cloud based data and machine learning pipelines Writing clear, maintainable Python, PySpark, and SQL to transform and prepare data Helping shape scalable data models that support analytics, machine learning, and product development Working closely with Product … improving data reliability, clarity, and long term platform health What we are looking for Experience using Python for data transformation, ideally with exposure to PySpark Confidence working with SQL and production data models, with an emphasis on clarity and reliability Experience working with at least one modern cloud data ...

junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). The Role As a Tech Lead , you will play a critical role in designing, building, and maintaining data pipelines and infrastructure that enable … junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data ...

treating data pipelines as production-grade software. A significant portion of the work involves designing and maintaining complex, test-driven Python data flows, with PySpark used as the execution engine rather than the primary focus. Strong Python fundamentals, testing discipline, and code quality are critical to success in this … strong emphasis on clean architecture, reusable Python modules, and testability Writing comprehensive unit tests and BDD tests (Behave), including mocking and patching Using PySpark to process large-scale datasets while keeping business logic in Python Creating, maintaining, and optimising Delta Lake tables for performance and reliability Building and running ...

support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability and resilience of existing … Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with production systems in a regulated ...

Services : S3, Redshift, Glue, Athena, Lake Formation, IAM Data Engineering/ETL: Building and optimizing ETL pipelines Data ingestion, transformation & orchestration using AWS Glue (PySpark/Python) Working with structured/semi-structured telecom datasets (CDRs, network logs, subscriber data) Data Lake Technologies: Expertise in Apache Iceberg table format … Candidate Profile 3 10+ years of experience in AWS Data Engineering/Support Engineering. Strong telecom domain understanding. Hands-on with Iceberg, Athena, Glue (PySpark), Python, Redshift, S3, ETL frameworks. Strong troubleshooting mindset and ability to work in 24 7 or rotational support environments (if required). ...

processing, and analytics • Designing scalable, secure, and high performance architectures using Databricks and distributed data systems • Building and overseeing robust data ingestion pipelines using PySpark, with a strong focus on reliability and accuracy • Ensuring end to end data quality from raw ingestion through to curated datasets used for reporting … wider regulatory requirements • Evaluating and introducing new technologies where they add clear value to the data ecosystem What you will bring • Deep expertise in PySpark and distributed data processing at scale • Strong hands on experience designing, building, and optimising Databricks based platforms • Advanced SQL skills including performance optimisation ...

Cleared Python Data Engineer Azure & PySpark Contract | £400£458 per day (Inside IR35) SC Clearance is essential We are looking for an experienced Data Engineer with strong Python skills to join a delivery-focused team working on complex data platforms in Azure. Although this is a data engineering role … work involves developing and maintaining complex, test-driven data flows. You will be responsible for building scalable data pipelines using Python and PySpark, optimizing Delta Lake storage, and deploying solutions in a containerised Azure environment. Quality, maintainability, and automated testing are key to this role. What youll be doing ...

At Flat Rock Technology, we build the future! Whether it's helping clients grow their teams through staff augmentation, developing software that scales businesses, protecting companies from cyber threats, or managing large operational teams as ...

days per week onsite What you'll be doing: Design, build, and optimise scalable data pipelines and architecture Work hands-on with SQL, PySpark, Databricks, Azure, and Data Lake to deliver high-performance solutions Translate business needs into technical requirements and data-driven solutions Ensure best practice in data … environment What you'll need: Proven experience as a Data Engineer or Senior Data Engineer with some team leadership responsibilities Strong technical expertise with PySpark, Databricks, Azure, and Data Lake Deep understanding of data architecture and ETL processes in cloud environments The ability to balance coding and technical delivery ...

days per week onsite What you’ll be doing: Design, build, and optimise scalable data pipelines and architecture Work hands-on with SQL, PySpark, Databricks, Azure, and Data Lake to deliver high-performance solutions Translate business needs into technical requirements and data-driven solutions Ensure best practice in data … environment What you’ll need: Proven experience as a Data Engineer or Senior Data Engineer with some team leadership responsibilities Strong technical expertise with PySpark, Databricks, Azure, and Data Lake Deep understanding of data architecture and ETL processes in cloud environments The ability to balance coding and technical delivery ...

Pyspark Engineer- (Data Engineering, AWS GLUE) SC Cleared OR EligibleStevenage (Hybrid) 2-3 days onsiteUp to £80,000High-impact programme - Revolutionary platformI am looking for a Pyspark expert to take the reins on a range of highly ambitious Data Migration projects supporting a range of truly high-impact … organisational and regulatory standards.Relevant exposure; Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Proficient-level SQL skills for complex query development, performance tuning, indexing, and data transformation across on-premise databases and AWS cloud environments. ...

Practice Lead – Defence Data Transformation Location: London/Hybrid Salary: £90,000 – £110,000 + Bonus & Benefits Palantir Foundry | Python | PySpark | Ontology | Data Engineering*Eligibility for clearance is required Join a £20m Programme Driving the Future of Defence Data Our client has just secured a £20 million programme … Stakeholders —translate technical concepts into operational impact. What We’re Looking For Essential: Palantir Foundry experience, ontology modelling, Agile leadership. Tech Stack: Python, SQL, PySpark, strong data engineering principles. Bonus Skills: Graph technologies (RDF, OWL, SPARQL), cloud platforms (AWS/Azure/GCP), integration tools (Airflow, Kafka, Elasticsearch). ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we're looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

days per week onsite What you’ll be doing: Design, build, and optimise scalable data pipelines and architecture Work hands-on with SQL, PySpark, Databricks, Azure, and Data Lake to deliver high-performance solutions Translate business needs into technical requirements and data-driven solutions Ensure best practice in data … environment What you’ll need: Proven experience as a Data Engineer or Senior Data Engineer with some team leadership responsibilities Strong technical expertise with PySpark, Databricks, Azure, and Data Lake Deep understanding of data architecture and ETL processes in cloud environments The ability to balance coding and technical delivery ...

support high variability Build and extend generative AI-enabled workflows as part of the core product Develop and maintain robust data pipelines using Python, PySpark, and distributed systems Balance near-term delivery with long-term platform maintainability and coherence Contribute to architectural decisions shaping the evolution of the platform … building platforms or products , not just one-off solutions Strong ability to turn ambiguous business problems into clean technical designs Solid foundations in Python, PySpark, and SQL Comfortable operating in a fast-moving startup environment Nice to Have Experience with modern data platforms (e.g. Foundry, Databricks, Snowflake, dbt, Airflow ...

support high variability Build and extend generative AI-enabled workflows as part of the core product Develop and maintain robust data pipelines using Python, PySpark, and distributed systems Balance near-term delivery with long-term platform maintainability and coherence Contribute to architectural decisions shaping the evolution of the platform … building platforms or products, not just one-off solutions Strong ability to turn ambiguous business problems into clean technical designs Solid foundations in Python, PySpark, and SQL Comfortable operating in a fast-moving startup environment Nice to Have Experience with modern data platforms (e.g. Foundry, Databricks, Snowflake, dbt, Airflow ...

price scraping outputs, mystery shopping insights) into practical pricing frameworks Work with Python-based pricing models and support their migration into scalable SQL/PySpark environments Translate complex analysis into clear, commercially relevant insight for non-technical stakeholders Act as a trusted interface between Pricing, Finance, Commercial teams, Category … closely with business stakeholders, not purely in a technical silo Comfort operating with imperfect or noisy data in real-world commercial environments Desirable Python, PySpark, or similar analytical/model-driven tools Exposure to B2B pricing, wholesale, distribution, or multi-site trading environments Understanding of ERP-driven pricing processes ...

Adapt and deploy the platform to customer-specific contexts and priorities Design and implement scalable generative AI-enabled workflows Execute complex data integrations using PySpark and distributed data technologies Collaborate closely with stakeholders to define success metrics and outcomes What We're Looking For 2–4+ years' experience … strong desire to work in such environments) Proven ability to translate ambiguous business requirements into clean, maintainable data pipelines Strong fundamentals in Python, PySpark, and SQL Ability to learn new technologies quickly and operate with a high degree of ownership Nice to Have Experience with dashboarding or data visualisation ...