51 to 69 of 69 PySpark Jobs in the UK excluding London

a best-in-class Lakehouse from scratch-this is the one. What You'll Be Doing Lakehouse Engineering (Azure + Databricks) Engineer scalable ELT pipelines using Lakeflow Declarative Pipelines , PySpark , and Spark SQL across a full Medallion Architecture (Bronze Silver Gold) . Implement ingestion patterns for files, APIs, SaaS platforms (e.g. subscription billing), SQL sources, SharePoint and SFTP using … growing data function. Tech Stack You'll Work With Databricks : Lakeflow Declarative Pipelines, Workflows, Unity Catalog, SQL Warehouses Azure : ADLS Gen2, Data Factory, Key Vault, vNets & Private Endpoints Languages : PySpark, Spark SQL, Python, Git DevOps : Azure DevOps Repos, Pipelines, CI/CD Analytics : Power BI, Fabric What We're Looking For Experience 5-8+ years of Data Engineering … with 2-3+ years delivering production workloads on Azure + Databricks . Strong PySpark/Spark SQL and distributed data processing expertise. Proven Medallion/Lakehouse delivery experience using Delta Lake . Solid dimensional modelling (Kimball) including surrogate keys, SCD types 1/2, and merge strategies. Operational experience-SLAs, observability, idempotent pipelines, reprocessing, backfills. Mindset Strong grounding More ❯

and implement scalable data pipelines that form the backbone of our Lakehouse platform, enabling accurate analytics, reporting, and regulatory compliance. You'll work with cutting-edge technologies including Databricks , PySpark , and Azure Data Factory , applying best practices in data engineering and governance to support this critical programme. Lead Databricks Engineer: Key Responsibilities Build and maintain Databricks pipelines (batch and … incremental) using PySpark and SQL. Orchestrate end-to-end workflows with Azure Data Factory . Develop and optimise Delta Lake tables (partitioning, schema evolution, vacuuming). Implement Medallion Architecture (Bronze, Silver, Gold) for transforming raw data into business-ready datasets. Apply robust monitoring, logging, and error-handling frameworks. Integrate pipelines with downstream systems such as Power BI . Collaborate … deliver consistent, well-documented datasets. Support deployments and automation via Azure DevOps CI/CD . Gather and refine requirements from business stakeholders. Lead Databricks Engineer: About You Strong PySpark development skills for large-scale data engineering. Proven experience with Databricks pipelines and workflow management. Expertise in Azure Data Factory orchestration. Solid knowledge of Delta Lake and Lakehouse principles. More ❯

Senior Data Engineer Salary: Up to £70,000 I am working with a forward-thinking organisation that is modernising its data platform to support scalable analytics and business intelligence across the Group. With a strong focus on Microsoft technologies and More ❯

Your new company This is a pivotal opportunity to join the Data and Innovation division of a large complex organisation leading the delivery of SAM (Supervisory Analytics and Metrics)-a transformative programme enhancing supervisory decision-making through advanced data and More ❯

Data Scientist Location: Hybrid (Greater Bristol Area) Salary: £54,000 Python - PySpark - Azure - Pandas - Scikit-learn - TensorFlow - PyStats - Data Science - Power BI We're supporting a growing, forward-thinking organisation in their search for an experienced Data Specialist. This is an exciting opportunity to join a dynamic team at a pivotal point in its growth, helping shape data strategy … MSc in AI/ML, Data Science, Mathematics, Computer Science or similar discipline. IoT Data Scientist (Data Science & Data Engineering) Location: Hybrid (Greater Bristol Area) Salary: £54,000 Python - PySpark - Azure - Pandas - Scikit-learn - TensorFlow - PyStats - Data Science - Power BI More ❯

ML systems, deploying LLMs, and operationalizing models in production. Key Responsibilities: Design, develop, and deploy ML, Deep Learning, and LLM solutions. Implement scalable ML and data pipelines in Databricks (PySpark, Delta Lake, MLflow). Build automated MLOps pipelines with model tracking, CI/CD, and registry. Deploy and operationalize LLMs , including fine-tuning, prompt optimization, and monitoring. Architect secure … . Mentor engineers, enforce best practices, and lead design/architecture reviews. Required Skills & Experience: 5+ years in ML/AI solution development. Recent hands-on experience with Databricks, PySpark, Delta Lake, MLflow . Experience with LLMs (Hugging Face, LangChain, Azure OpenAI) . Strong MLOps, CI/CD, and model monitoring experience. Proficiency in Python, PyTorch/TensorFlow, FastAPI More ❯

business challenges by adapting and deploying the STRATOS platform to their needs Design and implement scalable generative AI workflows using platforms like Palantir AIP Execute complex data integration using PySpark and other distributed technologies Collaborate directly with clients to understand context, priorities, and key outcome Requirements Minimum 2-4 years of solid experience in data engineering or analytics. Experience … scaling tech company, alternatively experience at a top-tier consultancy. The ability to translate complex and sometimes ambiguous business requirements into clean and maintainable data pipelines Excellent knowledge of PySpark, Python and SQL fundamentals The ability to get to grips with new technologies quickly What's Nice to Have** Experience in dashboarding tools, Typescript and API development Familiarity with More ❯

and cluster tuning to ensure cost-efficient, high-performing workloads. Implement data quality, lineage tracking, and access control policies aligned with Databricks Unity Catalogue and governance best practices. Develop PySpark applications for ETL, data transformation, and analytics, following modular and reusable design principles. Create and manage Delta Lake tables with ACID compliance, schema evolution, and time travel for versioned … data management. Integrate Databricks solutions with Azure services such as Azure Data Lake Storage, Key Vault, and Azure Functions. What We're Looking For Proven experience with Databricks, PySpark, and Delta Lake. Strong understanding of workflow orchestration, performance optimisation, and data governance. Hands-on experience with Azure cloud services. Ability to work in a fast-paced environment and deliver More ❯

one. What You'll Be Doing Designing end-to-end Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical decisions. Implementing CI/CD … projects with cutting-edge Microsoft technologies. What We're Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps/GitHub and Git version More ❯

one. What You'll Be Doing Designing end-to-end Microsoft Fabric architectures using the Medallion pattern. Building metadata-driven ingestion pipelines and optimising Delta Lake performance. Writing advanced PySpark/Spark SQL notebooks for large-scale transformations. Developing semantic models for enterprise reporting and enabling Direct Lake. Leading client workshops and guiding technical decisions. Implementing CI/CD … projects with cutting-edge Microsoft technologies. What We're Looking For Hands-on experience with Microsoft Fabric workloads: Lakehouse, Data Factory, Pipelines, Notebooks, Delta Lake, Eventstreams, Semantic Models. Advanced PySpark/Spark SQL skills and strong data engineering fundamentals. Integration experience with Dynamics 365, Dataverse, Business Central. CI/CD expertise with Azure DevOps/GitHub and Git version More ❯

and creating scalable AI workflows that bring real business impact. What you'll do: Deploy and customise a powerful data platform for global clients Build and optimise pipelines using PySpark, Python, and SQL Design scalable AI workflows with tools like Palantir Collaborate with client teams to deliver data-driven outcomes What we're looking for: 2-4 years' experience … in data engineering or analytics Hands-on with PySpark, Python, and SQL A proactive problem-solver who thrives in a fast-moving startup Excellent communication and stakeholder skills Why join: £50,000-£75,000 + share options Hybrid working (2-3 days/week in Soho) Highly social, collaborative culture with regular events Work alongside top industry leaders shaping More ❯

driven team of world-class technologists and business leaders working at the forefront of innovation and real-world impact. In this role, you will: Deliver complex data projects using PySpark and modern data tools Build scalable generative AI workflows using modern infrastructure Collaborate cross-functionally to ensure seamless delivery and adoption Drive innovation and continuous improvement across client engagements … To be successful in this role, you will have: Proven experience in data engineering or integration Strong proficiency in Python and PySpark Exposure to generative AI platforms or a passion for building AI-powered solutions Ability to lead client delivery in dynamic, fast-paced environments Familiarity with tools like Airflow, Databricks or DBT is a plus What's on More ❯

Data Engineer - Active SC, Databricks, PySpark Up to £475 per day (Inside IR35) Remote/London 6 months My client is an International Consultancy who are recruiting for an Data Engineer with Active Security Clearance (SC) and strong Databricks and Azure experience to deliver and optimise Data engineering solutions. Key requirements: Proven experience as a Data Engineer with Active … Security Clearance (SC) Strong experience with Databricks, PySpark and Delta Lake Expertise in Jobs & Workflows, cluster tuning, and performance optimisation Solid understanding of Data governance (Unity Catalog, Lineage, Access Policies) Hands-on with Azure services: Data Lake Storage (Gen2), Key Vault, Azure Functions Familiarity with CI/CD for Databricks deployments Strong troubleshooting in distributed Data environments Excellent communication More ❯

system. The successful candidate will play a critical role in supporting a broader initiative to migrate the ETL engine for the data set from Oracle PL/SQL to PySpark on Microsoft Fabric. Key Responsibilities: - Collaborate closely with the existing team to ensure a smooth transition of data during the migration process. - Optimise data pipelines for efficiency and performance. … migration initiatives. Qualifications: - Strong experience in data engineering roles, with a proven track record of data migration projects. - Familiarity with Oracle and Microsoft Fabric environments is essential. - Proficiency in PySpark is required. - Excellent analytical and problem-solving skills, with attention to detail. - Strong communication skills to effectively collaborate with team members and stakeholders. More ❯

About the RoleWe are looking for a Python Data Engineer with strong hands-on experience in Behave-based unit testing, PySpark development, Delta Lake optimisation, and Azure cloud services. This role focusses on designing and deploying scalable data processing solutions in a containerised environment, emphasising maintainable, configurable, and test-driven code delivery.Key Responsibilities Develop and maintain data ingestion, transformation … Azure Functions for serverless transformation logic Azure Key Vault for secure credential management Azure Blob Storage for data lake operations What We're Looking For Proven experience in Python, PySpark, and Delta Lake. SC Cleared Strong knowledge of Behave for test-driven development. Experience with Docker and containerised deployments. Familiarity with Azure cloud services and data engineering best practices. More ❯

Reed Technology is delighted to be partnering with an innovative, cutting-edge company based in Aberdeen, currently seeking to permanently onboard a Python Developer. The successful candidate will join a dynamic product development team, contributing to the full software development More ❯

Seeking a hands-on data platform architect/engineer to reverse-engineer a legacy solution (currently on a VM) and migrate it to Microsoft Fabric. The goal is to stabilize critical data processes and lay the groundwork for a modular More ❯

Seeking a hands-on data platform architect/engineer to reverse-engineer a legacy solution (currently on a VM) and migrate it to Microsoft Fabric. The goal is to stabilize critical data processes and lay the groundwork for a modular More ❯

team for testing of data demonstrating real experience in technology and processes • Technology experience o JMeter – for API testing (Load and Performance testing experience not needed) o Groovy o Pyspark – Should be able to write scripts for data processing o Data Lakes o Kafka Streamsets o AWS cloud Requirements • Overall Exp years, Relevant exp years. • Excellent experience in testing … team for testing of data demonstrating real experience in technology and processes • Technology experience o JMeter – for API testing (Load and Performance testing experience not needed) o Groovy o Pyspark – Should be able to write scripts for data processing o Data Lakes o Kafka Streamsets o AWS cloud More ❯