301 to 325 of 344 PySpark Jobs

improving data pipelines. Working with Databricks to improve the existing data platform. Getting hands-on with data transformation and modelling. Using Python, SQL and PySpark to deliver data solutions. Supporting the migration from QuickSight to Power BI. Working closely with teams across Finance, Pricing, Regulatory and Compliance. Helping … small team where everyone gets involved and priorities can change. Insurance experience would be useful, as would any Power BI, AWS and PySpark experience. What's on offer? £80,000–£85,000 salary. Generous bonus scheme 25 days' holiday, increasing to 30 with length of service. Hybrid working, with ...

data exploration skills Experience of working in Agile methodology Good knowledge and hands-on experience of developing Stored Procedures Experience of working with PySpark Notebooks for data ingestion and provision Practical experience of improving data quality and efficiency in data pipelines Solid understanding of DevOps and CI/… Understanding of compliance and lifecycle of data management Desirable Skills: Coding experience in any relevant language (Python, Pyspark etc.) Experience working with other Azure technologies such as Fabric, Power Automate Experience of working with non-relational databases (CosmosDB, MongoDB, etc.) and handling unstructured data in ETL pipelines Our values ...

develop, and maintain cloud‐native data solutions that ensure data is accurate, accessible, and trusted. Designing and developing scalable data pipelines using Python and PySpark Building and optimising data workflows using AWS Glue, Athena, Redshift , and related AWS services Supporting modern lakehouse architectures, including Apache Iceberg Working with relational … solutions. Essential experience Experience as a Data Engineer, ETL Developer, Data Platform Engineer, or a similar role Strong hands‐on experience with Python and PySpark Experience with AWS Glue, Amazon Athena, Amazon Redshift , and cloud‐based data platforms Strong SQL skills and experience working with both relational and NoSQL ...

maintainable solutions. What you'll be doing: Take ownership of designing, building, and maintaining scalable, production-ready data pipelines using Microsoft Fabric, Python, and PySpark, with minimal supervision Make sound technical decisions on pipeline architecture, data modelling, and tooling, weighing up trade-offs in performance, cost, and maintainability Mentor … domain-driven modelling) - this is the load‐bearing, provider‐agnostic skill underneath almost all of our pipeline work Have strong proficiency in Python and PySpark for batch data processing, with a track record of writing production‐grade, well‐tested code (this role is focused on batch/ELT pipeline ...

Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common IDs, Golden Records, entity … entity matching, deduplication, and survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity ...

Experience: 610+ Years Key Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common … entity matching, deduplication, and survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity ...

Senior Data Engineer – Contract £600 per day | Outside IR35 | Fully Remote | 2-Month Contract | Start: 1st November We are looking for a highly experienced Senior Data Engineer to support a critical data platform migration project ...

DV Cleared Data Engineer Location: West Midlands - Hybrid working Rate: Up to £600 per day Inside IR35 Contract: Until 31 March 2027, with extensions expected Clearance: Active DV Clearance essential The Role We are seeking ...

Databricks Model Serving, monitoring compute resource allocation and cold-start latency. Large-Scale Data & Feature Engineering: Design and implement scalable data transformation workflows using PySpark and maintain reusable features using the Databricks Feature Store. Governance & Security: Enforce end-to-end data and model lineage, role-based access control (RBAC … Advanced proficiency in Python (OOP, typing, design patterns, testing frameworks) and advanced analytical SQL. Distributed Computing - Deep hands-on knowledge of Apache Spark/PySpark, including distributed query optimization, partition tuning, memory management, and debugging skewed datasets. ML & MLOps - Strong experience with MLflow (Tracking, Model Registry, Projects), plus standard ...

helping to raise the bar on engineering practice. The foundations still matter: this is a data engineering role at heart, built on Python, PySpark, and platforms like Databricks, Snowflake, or AWS. What you'll be doing: Take ownership of designing, building, and maintaining the production data and feature pipelines … that catches drift and degradation before a client does Build and maintain the batch and ELT pipelines the ML work sits on, using Python, PySpark, and cloud data services (e.g., AWS Glue, EMR, S3, Redshift, Athena, or the Databricks/Snowflake equivalents), making sound decisions on architecture, data modelling ...

helping to raise the bar on engineering practice. The foundations still matter: this is a data engineering role at heart, built on Python, PySpark, and platforms like Databricks, Snowflake, or AWS. What you'll be doing: Take ownership of designing, building, and maintaining the production data and feature pipelines … that catches drift and degradation before a client does Build and maintain the batch and ELT pipelines the ML work sits on, using Python, PySpark, and cloud data services (e.g., AWS Glue, EMR, S3, Redshift, Athena, or the Databricks/Snowflake equivalents), making sound decisions on architecture, data modelling ...

helping to raise the bar on engineering practice. The foundations still matter: this is a data engineering role at heart, built on Python, PySpark, and platforms like Databricks, Snowflake, or AWS. What you'll be doing: Take ownership of designing, building, and maintaining the production data and feature pipelines … that catches drift and degradation before a client does Build and maintain the batch and ELT pipelines the ML work sits on, using Python, PySpark, and cloud data services (e.g., AWS Glue, EMR, S3, Redshift, Athena, or the Databricks/Snowflake equivalents), making sound decisions on architecture, data modelling ...

helping to raise the bar on engineering practice. The foundations still matter: this is a data engineering role at heart, built on Python, PySpark, and platforms like Databricks, Snowflake, or AWS. What you'll be doing: Take ownership of designing, building, and maintaining the production data and feature pipelines … that catches drift and degradation before a client does Build and maintain the batch and ELT pipelines the ML work sits on, using Python, PySpark, and cloud data services (e.g., AWS Glue, EMR, S3, Redshift, Athena, or the Databricks/Snowflake equivalents), making sound decisions on architecture, data modelling ...

influence delivery standards and contribute to reusable frameworks and accelerators. What You’ll Be Doing Building production-grade data pipelines with Databricks, PySpark and SQL Developing Lakehouse architectures using Delta Lake, Unity Catalog and medallion frameworks Gathering requirements and providing technical guidance to client teams Embedding CI/… team Modernising legacy data platforms to unlock more value from client data estates What You’ll Bring Commercial Databricks delivery experience Strong hands-on PySpark and SQL Solid grounding in data engineering, data modelling and Lakehouse architecture Experience on Azure and/or AWS Source control, CI/ ...

week. Inside IR35 via umbrella. Start – asap. Must be eligible to work in the UK. Must Have Skills Strong hands-on Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting Workflow orchestration and dependency management Idempotency, retry/restart and failure-recovery patterns CI/… manual intervention Stable QA production deployment process Runbooks and operational handover Reduced runtime and production failures Key Skills Must Have: Strong hands-on Databricks, PySpark, Python and SQL Production pipeline engineering and troubleshooting Workflow orchestration and dependency management Idempotency, retry/restart and failure-recovery patterns CI/ ...

Snowflake - Warehouses, Streams/Tasks, Snowpipe, Snowpark, UDFs/Stored Procedures, Time Travel and Cloning. Snowpark and end-to-end pipeline development. Experience with PySpark, Polars or Dask is also valuable. Python, including testing, build tooling and engineering best practices. SQL, including Snowflake-optimised SQL. dbt, including Jinja templating … champion better ways of working. Nice to Have Experience with Docker, Java/Scala, Kafka, Pulsar, Azure Event Hub, Collibra, Apache Iceberg, Apache Arrow, PySpark or AI engineering technologies such as Snowflake AI, LangChain, LlamaIndex or Pinecone would be advantageous. This is a great opportunity to join a collaborative ...

JOB ID: FEQ427R333 Location: The customer is based in Leeds or Manchester, United Kingdom Industry: Enterprise Retail or CPG (UKI Enterprise) - The customer is based in Leeds. You will be visiting the client or based ...

Test Automation Engineer - SC Cleared £300 per day - Outside IR35 6-month contract with scope to extend Central London - 3x days a week We're seeking a Test Automation Engineer to automate testing for data ...

systems, databases and third‐party APIs, shaping it into trusted datasets used across the business. You’ll work with Microsoft Fabric, SQL, Python/PySpark and Power BI, ensuring data quality, lineage and scalable models that support dashboards and decisions across teams. #J-18808-Ljbffr ...

organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements: Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong … Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously in a fast-moving environment Experience with Databricks Apps, Lakebase, Airflow, Azure Data Factory, monitoring platforms or financial services would be beneficial. ...

Inside IR35 via umbrella Remote Must be eligible to work in the UK Start ASAP Must have: Strong hands-on Databricks, SQL, Python/PySpark ETL/ELT Data-pipeline migration Legacy-to-cloud modernisation Production troubleshooting Dependency analysis Pipeline repointing Error handling and recovery Data validation/reconciliation … Clear handover into BAU CIS/SCI Integration & Post-Migration Stabilisation Data Engineer Key Skills: Must Have: Strong hands-on Databricks, SQL, Python/PySpark ETL/ELT Data-pipeline migration Legacy-to-cloud modernisation Production troubleshooting Dependency analysis Pipeline repointing Error handling and recovery Data validation/reconciliation ...

Technical Skills You'll have experience with some of the following: Cloud & Data Platforms: AWS, Azure, Google Cloud Platform, Databricks Programming Languages: Python, R, PySpark Agentic & GenAI Platforms: Microsoft Copilot Studio, OpenAI GPT-5 Agents, UiPath, Adept AI, Orby AI, Beam AI If you've held any of these … Specialist, Python Developer, AI Transformation Consultant, Databricks Consultant, Azure AI Consultant, AWS Data Scientist, GCP Data Consultant, Agentic AI Consultant, OpenAI GPT Specialist, PySpark Developer. Deerfoot Recruitment Solutions Ltd is a leading independent tech recruitment consultancy in the UK. For every CV sent to clients, we donate ...

contribute across the length and breadth of a project Proficiency in one or more of the following: Java/Spring Boot, Python/PySpark, GraphQL, or mobile development frameworks, with openness and aptitude to work across others as needed Experience with cloud platforms (preferably AWS) and working knowledge … financial services industry and their IT systems Preferred qualifications, capabilities, and skills Experience with high-volume batch and real-time data processing using PySpark or similar frameworks, including performance tuning and troubleshooting of large-scale data pipelines In-depth knowledge of relational and distributed databases (e.g., Oracle, Databricks) with ...

forecasting, gradient boosting, clustering, NLP and Bayesian methods Strong MLOps: orchestration, experiment tracking, hyperparameter tuning and automated retraining pipelines Object-oriented Python, Polars or PySpark, and data versioning (e.g. DVC), plus experience designing data storage Hands-on experience with at least two of AWS, Azure and GCP, including deployment … solid grasp of CI/CD, testing and containerisation Nice to have A Masters or PhD Experience with parallel processing frameworks such as PySpark or Ray Serverless deployments (e.g. Fargate, Lambda) and infrastructure automation with tools like Terratest or Ansible Whats on offer Salary: Up to £113,000 Hybrid ...

sets Programming experience (focus on machine learning): SQL and Python’s Data Science stack; good knowledge of at least one big data framework (e.g., PySpark, Hive, Hadoop) is a plus. R, SPSS, and SAS are considered nice‐to‐have Strong understanding of machine learning methods and experience applying them … into production or applied business environments is a plus While we advocate using the right tech for the right task, we often leverage: Python, PySpark, the PyData stack, SQL, Airflow, Databricks, Kedro (our open‐source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions such ...