76 to 100 of 101 PySpark Jobs in the UK

able to develop the prototypes of the models, and do some feature engineering. The environment looks a bit like this 👇 👉 Python (pandas, numpy, scipy, PySpark) 👉 SQL 👉 AWS or GCP (BigQuery/RedShift/Snowflake ...

expertise in data processing within a cloud-based data platform. Junior Data Engineer duties: Develop, test, and maintain scalable data pipelines using Python and PySpark within an Azure Databricks development environment Clean, aggregate, and transform complex datasets to meet business requirements. Assist with data quality checks and support data … Junior Data Engineer, you will have: Strong proficiency in Python, particularly for data manipulation and automation scripts; experience in libraries such as pandas, PySpark, numpy, pyodbc or associated would be beneficial Hands-on experience (or academic project experience) using PySpark. An understanding of ETL/ELT processes and/ ...

Defining and enforcing engineering standards across code quality, testing, CI/CD, IaC, and observability Building and optimising ELT/ETL pipelines using Glue (PySpark) , Lambda, Step Functions, and event-driven patterns Championing security-by-design in collaboration with security and assurance teams (IAM, KMS, encryption, auditability) Implementing … across data analytics and data engineering domains Strong hands-on experience with: Amazon S3 (data lake patterns, partitioning, lifecycle management) AWS Glue (Jobs, Crawlers, PySpark, orchestration) Amazon Redshift (data modelling, performance tuning, WLM, Spectrum) Advanced Python (PySpark) and SQL skills Experience implementing CI/CD pipelines and Infrastructure ...

junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). The Role As a Tech Lead , you will play a critical role in designing, building, and maintaining data pipelines and infrastructure that enable … junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data ...

hypothesis testing, confidence intervals, error rate control, and handling biases/confounders Proficiency in at least one language for statistical analysis (Python, R, or PySpark), with the ability to develop and implement simulation or analysis frameworks Track record of clear, effective explanation of statistical concepts and results ...

Job Responsibilities: Payments and File Format Expertise Deep understanding of payment processing workflows, especially H2H and SWIFT. Hands-on experience with legacy formats (IDOC, EDIFACT) as well as ISO 20022 and GFF. Ability to interpret ...

support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability and resilience of existing … Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with production systems in a regulated ...

innovative data platform. You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

working with Databricks** You'll play a pivotal role in designing, building, and optimising their Azure-based Data Lakehouse, with a focus on Databricks, PySpark, Spark SQL, and Azure Data Factory. This isn't just about coding - you'll also provide architectural guidance, mentor engineers, and ensure solutions … Develop automation for cluster management, integration runtimes, and networking Lead architectural design and ensure platform scalability, reliability, and governance Write efficient, maintainable code in PySpark, Python, and SQL Implement CI/CD pipelines and cloud infrastructure via Terraform/ARM Collaborate with data engineers, architects, and business stakeholders Mentor ...

Services : S3, Redshift, Glue, Athena, Lake Formation, IAM Data Engineering/ETL: Building and optimizing ETL pipelines Data ingestion, transformation & orchestration using AWS Glue (PySpark/Python) Working with structured/semi-structured telecom datasets (CDRs, network logs, subscriber data) Data Lake Technologies: Expertise in Apache Iceberg table format … Candidate Profile 3 10+ years of experience in AWS Data Engineering/Support Engineering. Strong telecom domain understanding. Hands-on with Iceberg, Athena, Glue (PySpark), Python, Redshift, S3, ETL frameworks. Strong troubleshooting mindset and ability to work in 24 7 or rotational support environments (if required). ...

treating data pipelines as production-grade software. A significant portion of the work involves designing and maintaining complex, test-driven Python data flows, with PySpark used as the execution engine rather than the primary focus. Strong Python fundamentals, testing discipline, and code quality are critical to success in this … strong emphasis on clean architecture, reusable Python modules, and testability Writing comprehensive unit tests and BDD tests (Behave), including mocking and patching Using PySpark to process large-scale datasets while keeping business logic in Python Creating, maintaining, and optimising Delta Lake tables for performance and reliability Building and running ...

doing Design, build and deploy algorithms/data models supporting pricing, forecasting and optimisation use cases in production Develop and optimise distributed Spark/PySpark batch pipelines for large-scale data processing Write production-grade Python workflows implementing complex, explainable business logic Work with Databricks for job execution, orchestration … delivering production algorithms/data models (forecasting, pricing, optimisation or similar) Strong Python proficiency and modern data stack exposure (SQL, Pandas/NumPy + PySpark; Dask/Polars/DuckDB a bonus) build, schedule and optimise Spark/PySpark pipelines in Databricks (Jobs/workflows, performance tuning, production ...

models. Develop time-series forecasting models for demand, liquidity, transaction volume and customer behaviour prediction. Develop real-time and batch ML pipelines using Python, PySpark and modern data stacks. Build and optimise models using high-volume, low-latency data (transactions, behavioural signals, device data). Implement MLOps best practices … including CI/CD, model versioning, monitoring and retraining pipelines. Work with streaming technologies such as Kafka. Proficient in Python, SQL, PySpark, CI/CD, and Cloud (GCP/AWS/Azure). Excellent communication skills with both technical and business stakeholders. If this sounds like the right opportunity ...

data science/analytics with expertise in Security, Anti-Cheat, Trust & Safety, or related domains. Strong SQL proficiency with large-scale databases. Advanced Python, PySpark, and data science tools (numpy, pandas, scikit-learn) in cloud production environments. Required expertise in one: graph analytics for coordinated behavior, NLP for content … analytics for coordinated behavior detection. Published research or thought leadership in Trust & Safety, content moderation, or online community safety. Experience with big data technologies (PySpark) and cloud platforms (AWS, GCP, Azure, Databricks). Multilingual capabilities or experience with non-English content moderation challenges. HOW TO APPLY Please apply with ...

Pyspark Engineer- (Data Engineering, AWS GLUE) SC Cleared OR EligibleStevenage (Hybrid) 2-3 days onsiteUp to £80,000High-impact programme - Revolutionary platformI am looking for a Pyspark expert to take the reins on a range of highly ambitious Data Migration projects supporting a range of truly high-impact … organisational and regulatory standards.Relevant exposure; Strong hands-on experience with ETL processes and tools (Talend, Informatica, Matillion, Pentaho, MuleSoft, Boomi) or scripting using Python, PySpark, and SQL. Proficient-level SQL skills for complex query development, performance tuning, indexing, and data transformation across on-premise databases and AWS cloud environments. ...

Our client is an industry leading market-research company who use data-driven insights to provide unique and actionable data for some of the world's most recognised brands. Over the last 12 months they ...

RealityMine has been a pioneer in delivering data driven insights to the world's largest brands for over a decade. Our platform provides unique data solutions to our clients enabling them to make strategic, informed ...

generative AI applications. Perform Exploratory Data Analysis (EDA), data mining, and visualization to extract insights. Design and implement Big Data solutions using Hadoop, Spark, PySpark, and DataLake architectures. Develop and deploy ML models (supervised, unsupervised, tree-based, ensemble methods). Implement deep learning architectures (CNN, RNN, LSTM) using TensorFlow … systems. Required Skills: Analytical Tools: EDA, Data Mining, Visualization (Plotly, Matplotlib, Seaborn) Statistical & Multivariate Analysis Big Data: Hadoop, MapReduce, HDFS, DataBricks, Spark, PySpark DataLake Architecture, AWS Redshift, Kinesis, EMR Machine Learning Supervised Models: Naïve Bayes, Logistic Regression, SVM, Linear Regression, KNN Tree-Based Models: Decision Trees, Random Forest, Gradient ...

pricing models with advanced data science techniques using WTW Radar and Emblem. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Create innovative data solutions and present actionable insights to stakeholders. Communicate results clearly to key decision-makers … Street Pricing. What we're looking for: Experience in general insurance pricing with strong modelling skills. Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software is preferred. A quantitative degree in Mathematics ...

price scraping outputs, mystery shopping insights) into practical pricing frameworks Work with Python-based pricing models and support their migration into scalable SQL/PySpark environments Translate complex analysis into clear, commercially relevant insight for non-technical stakeholders Act as a trusted interface between Pricing, Finance, Commercial teams, Category … closely with business stakeholders, not purely in a technical silo Comfort operating with imperfect or noisy data in real-world commercial environments Desirable Python, PySpark, or similar analytical/model-driven tools Exposure to B2B pricing, wholesale, distribution, or multi-site trading environments Understanding of ERP-driven pricing processes ...

price scraping outputs, mystery shopping insights) into practical pricing frameworks Work with Python-based pricing models and support their migration into scalable SQL/PySpark environments Translate complex analysis into clear, commercially relevant insight for non-technical stakeholders Act as a trusted interface between Pricing, Finance, Commercial teams, Category … closely with business stakeholders, not purely in a technical silo Comfort operating with imperfect or noisy data in real-world commercial environments Desirable Python, PySpark, or similar analytical/model-driven tools Exposure to B2B pricing, wholesale, distribution, or multi-site trading environments Understanding of ERP-driven pricing processes ...

champion best practice methods; upskill the team through coaching and code reviews. Query and engineer large datasets (e.g., Python/R/SQL/PySpark) on modern platforms (e.g., Azure Databricks). Communicate pricing proposals and insights clearly and persuasively to stakeholders at all levels. Collaborate with Modelling, Optimisation … insurance pricing. Strong technical expertise in predictive modelling techniques (e.g. GLMs and machine learning models). Proficiency in programming languages such as R, Python, PySpark, SAS, or SQL. Good knowledge of Azure Databricks and modern data technologies. Experience with WTW Radar software and optimisation tools. A quantitative degree ...

solutions. GenAI Implementation: Hands-on deployment of Azure OpenAI, Vector Databases, and RAG patterns. Data Engineering: Oversee large-scale data processing using Python and PySpark to support AI model fine-tuning. Leadership: Manage technical delivery teams and deliver upskilling sessions to the customer. Stakeholder Management: Gain consensus on complex … roadmaps with diverse client groups. Technical Skills & Experience Core AI: Azure OpenAI, Prompt Engineering, Cognitive Search, Form Recognizer. Data/Coding: Expert-level Python, PySpark, and Vector DB integration. Ops: Practical knowledge of MLOps and LLMOps lifecycles. Security: Must hold active UK Government SC Clearance. Certifications Mandatory ...

Role Details Role/Job Title: Python/PySpark, Scala Developer Location: Glasgow (5 days onsite) Special Working Conditions (travel, on call etc.): 5 days onsite Glasgow city center Overview As a Python Lead Software Engineer With TATA Consultancy Services , you will play a crucial role in an agile … scalable manner. Key Responsibilities (Up to 10, Avoid repetition) Formal training or certification on software engineering concepts and advanced applied experience Advanced in Python & PySpark or Scala Experience in API development and exposure to Python Application development Hands-on practical experience delivering system design, application development, testing, and operational ...

building common functionality for distributed product teams. Early work will focus on: Abstracting existing codebases into shared Python libraries Developing data-driven solutions using PySpark and DataFrames Building and extending Python-based microservices using Azure Functions Creating scalable, reusable services that solve common challenges across teams Beyond the initial … essential, as youll be working closely with engineers and technical leads across five time zones . Essential skills Strong commercial experience with Python PySpark and data-frame based processing Solid SQL capability Experience working with Azure infrastructure Good understanding of containers, microservices, and functional design patterns Comfortable working ...