1 to 25 of 32 Permanent PySpark Jobs in the UK

addressing any concerns and ensuring incidents are effectively managed. What you'll need to succeed Knowledge/Qualifications - Expert-level SQL and Python (ideally PySpark). - A strong understanding of DataSecOps; experience with PII handling, data obfuscation, and RBAC (Role-Based Access Control). - Proven ability to design data ...

contribute to long-term platform innovation Required Experience Strong experience designing enterprise data platforms within Microsoft Azure environments Deep expertise with Azure Databricks, Spark (PySpark/Scala), Delta Lake, and Lakehouse architecture Strong hands-on experience with Snowflake architecture and optimisation Experience with Azure Data Factory, ADLS Gen2, Synapse ...

engagement skills, including requirements gathering, expectation management and delivery planning. Desirable Knowledge and practical experience of hospital systems and datasets (where relevant locally). PySpark experience. Experience with NLP and/or computer vision techniques (where relevant to Trust priorities). Experience using deep learning frameworks such as TensorFlow ...

with Databricks SQL endpoints for data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/ ...

programming, concurrency, multithreading) Expertise in RESTful API design, OpenAPI/Swagger documentation and API lifecycle best practices Hands-on experience with ML libraries: PyTorch, PySpark, scikit-learn, Hugging Face Transformers Proficiency in AI tooling: Azure ML, Databricks, MLflow, LangChain, LangGraph Proven experience with Git, unit testing, containerisation and agile ...

data providers-ensuring data is reliable, auditable, and delivered at scale. Data Engineer Responsibilities Act as the subject matter expert across ADF, Databricks, Python, PySpark, and Spark SQL Manage the daily ingestion and processing of raw data into enterprise data platforms Collaborate with the wider team to define data ...

Hands-on experience with Microsoft Purview • Strong Azure Data Platform experience: ADLS Gen2, Azure Databricks, Data Factory • Hands-on data engineering experience (Spark/PySpark, Delta Lake) • Knowledge of security and compliance controls, GDPR, PII/DLP, RBAC/ABAC • Experience leading Agile teams and managing risks, dependencies ...

Databricks Defining data models and Databricks architecture standards Establishing data governance and security frameworks Aligning data architecture with enterprise cloud strategy Supporting development using PySpark, SQL and Python within Databricks Working within CI/CD and DevOps environments (Git-based) About You: Strong technical background in Databricks and Azure ...

Looking For: Proven experience working with Palantir Foundry in production environments Strong engineering background with expertise in at least one of: Python, SQL, PySpark, or TypeScript Strong understanding of modern data architectures & distributed systems Ability to work in fast-paced, ambiguous environments Excellent stakeholder engagement and communication skills Experience ...

query performance Strong analytical and problem-solving skills Desirable Skills Knowledge of modern data architectures Experience with SQL, SSRS, SSIS and SSAS Experience using PySpark Familiarity with KQL Experience with data cataloguing tools Exposure to C# What you'll get in return Competitive Salary - 52,750 with a generous ...

ideally in a multi-site business environment Strong proficiency in Python and SQL, with hands-on experience in Spark/PySpark and modern table formats (e.g. Delta Lake, Iceberg, Hudi) Expertise in AWS data services (e.g. S3, Glue, Redshift, Lambda, Step Functions, Kinesis) and infrastructure-as-code (Terraform ...

join their Defence & National Security team. In order to be successful, you will have the following experience: Experienced in the Python data stack (PySpark, numpy, pandas, scikit-learn, etc.) Experienced within AWS-based infrastructure and services Excellent communication skills, able to work with customers to shape requirements Able ...

making Your Skills and Experience Strong commercial experience as a data engineer in modern, cloud based environments Advanced Python and SQL, with experience using PySpark or similar distributed processing frameworks Experience building and maintaining data pipelines in AWS using services such as S3, Glue or EKS Hands on experience ...

Kubernetes Experience in creating production grade coding and SOLID programming principles, including test-driven development (TDD) approaches Experience in programming languages (e.g. Python, PySpark, R, SAS, SQL) Experience in source-control software, e.g., GitHub Proficient at communicating results in a concise manner both verbally and written Experience in data ...

qualification in Computer Science, AI, Data Science, or related fields. Strong knowledge of AI, machine learning, deep learning, and probabilistic modeling. Proficient in Python, PySpark, data modeling, and microservices. Experience in MLOps, AI lifecycle management, and LLM orchestration on cloud platforms (AWS, GCP, Azure). Familiar with DevOps tools ...

qualification in Computer Science, AI, Data Science, or related fields. Strong knowledge of AI, machine learning, deep learning, and probabilistic modeling. Proficient in Python, PySpark, data modeling, and microservices. Experience in MLOps, AI lifecycle management, and LLM orchestration on cloud platforms (AWS, GCP, Azure). Familiar with DevOps tools ...

qualification in Computer Science, AI, Data Science, or related fields. Strong knowledge of AI, machine learning, deep learning, and probabilistic modeling. Proficient in Python, PySpark, data modeling, and microservices. Experience in MLOps, AI lifecycle management, and LLM orchestration on cloud platforms (AWS, GCP, Azure). Familiar with DevOps tools ...

qualification in Computer Science, AI, Data Science, or related fields. Strong knowledge of AI, machine learning, deep learning, and probabilistic modeling. Proficient in Python, PySpark, data modeling, and microservices. Experience in MLOps, AI lifecycle management, and LLM orchestration on cloud platforms (AWS, GCP, Azure). Familiar with DevOps tools ...

qualification in Computer Science, AI, Data Science, or related fields. Strong knowledge of AI, machine learning, deep learning, and probabilistic modeling. Proficient in Python, PySpark, data modeling, and microservices. Experience in MLOps, AI lifecycle management, and LLM orchestration on cloud platforms (AWS, GCP, Azure). Familiar with DevOps tools ...

understanding of DAG (Directed Acyclic Graphs) and their role in pipeline orchestration It would be nice if you could bring (desirables): Professional experience with PySpark, MySQL, Docker, Kubernetes Experience with best-practice pipeline orchestration software (Airflow, Dagster). Experience migrating code from prototype notebooks to .py, while maintaining logic ...

impact. Key Responsibilities Design and deliver end-to-end technical solutions across complex data environments Build and maintain applications using technologies such as Python, PySpark, TypeScript, and React Develop AI-driven workflows and operational tooling Work directly with client stakeholders in fast-paced delivery environments Present demos, gather feedback ...

TensorFlow Serving, TorchServe). Familiarity with ML lifecycle tools such as MLflow, Kubeflow, or Azure ML Pipelines. Experience working with distributed data processing (e.g., PySpark) and SQL. Understanding of software engineering best practices, including version control (Git). Knowledge of CI/CD principles in ML environments. Experience with ...

migration initiatives, including consolidation of legacy data, medallion architecture design, schema evolution, and cutover planning Provide hands-on data engineering expertise, building and optimising PySpark pipelines Ensure data is secure, high-quality, accessible, and structured for maximum business value Drive best practice in data governance, cataloguing, lineage, and standards … reporting teams Ensure compliance with data protection regulations and internal policies What you'll bring Strong hands-on experience in data engineering, including PySpark and production data pipelines Experience with Microsoft Fabric, Azure Synapse, or similar platforms (e.g. Databricks, AWS) Proven ability to design and manage scalable data pipelines ...

Collaborative office environment with regular social and charity events And Much More!!! DATA ENGINEER - ESSTENTIAL SKILLS Strong experience with Databricks and Apache Spark (PySpark or Scala) Experience building streaming and batch pipelines using Spark Structured Streaming Hands-on experience with Kafka (MSK) and real-time data ingestion Strong understanding … submit (subject to required skills) your application to our client in conjunction with this vacancy only. KEY SKILLS Data Engineering, Databricks, Apache Spark, PySpark, SQL, Python, Kafka, AWS, Delta Lake, ETL, Data Pipelines, CI/CD, Big Data ...

help deliver reliable, high-quality data to support business insights. Key Responsibilities Build and maintain scalable ETL/ELT pipelines using Python and PySpark Ingest and integrate data from multiple sources including trading, finance, and operational systems Work within a Databricks Lakehouse architecture to transform and deliver data Optimise … stakeholders to ensure data is accessible and usable Key Skills & Experience Strong experience in Data Engineering (3–6 years) Proficiency in Python and PySpark for large-scale data processing Hands-on experience with Databricks (Delta Lake, Workflows, Lakehouse architecture) Solid knowledge of AWS services (e.g. S3, Kinesis, Lambda ...