26 to 37 of 37 PySpark Jobs in the City of London

quality and structure of data Translate business needs into clear technical plans Experience required: Strong skills in data modelling SQL with either Python or PySpark Databricks or similar cloud data platform Experience communicating with both technical and non-technical users Leadership or mentoring experience Insurance or regulated sector experience ...

models and manages controlled, production-grade releases to ensure platform stability, scalability, and trusted analytics consumption. Key Responsibilities Design, govern, and operationalize Fabric pipelines, PySpark transformations, and Lakehouse-based Medallion architecture. Translate business requirements and approved data roadmaps into executable technical designs, sprint backlogs, and production-grade delivery plans. ...

/d Contract length: 6 months Base Location: London - 3/d a week Skills required: - Azure/Azure Data bricks - ADF/ETL - Pyspark/Scala Experience: Experience working on large data sets, and complex data pipelines click apply for full job details ...

/d Contract length: 6 months Base Location: London - 3/d a week Skills required: - Azure/Azure Data bricks - ADF/ETL - Pyspark/Scala Experience: Experience working on large data sets, and complex data pipelines. Understanding of Data Architecture and Design, and Data pipeline optimisation. Proven ...

Title: Azure Databricks Data Engineer Primary skills: Advanced SQL, Azure Databricks, Azure Data Factory, Azure Datalake. Secondary skills: Azure SQL, PySpark, Azure Synapse. Experience: 10+ Years of Experience Hybrid - 3 Days/Week onsite is MUST About the job We are looking for an experienced Databricks Data Engineer … ingestion and transformation. • Collaborate with Azure stack modules like Data Lakes and SQL DW to handle large volumes of data. • Write SQL, Python, and PySpark code to meet data processing and transformation needs. • Understand business requirements and create data flow processes that meet them. • Develop mapping documents and transformation ...

junior & Mid Data Engineers on the best practices when building and managing data infrastructure, including data lakes, warehouses, and distributed processing systems (e.g., PySpark, Hadoop). Data Preparation: Collaborate with data scientists to prepare and transform raw data into formats suitable for machine learning, including feature engineering and data … inclusive team environment that encourages continuous learning and improvement. Your Profile Essential skills/knowledge/experience: Knowledge of machine learning frameworks (e.g., PySpark, PyTorch ) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

Job Title: Data Engineering Lead Department: Technology & Data Location: London/Hybrid (4 days onsite) Salary: up to £110,000 DOE Gravitas is working with a London-based Investment Management company, which is looking to ...

optimise CI/CD pipelines in Azure DevOps to automate deployment of workspaces, Unity Catalog, networking, and security. Work with Databricks (Spark/Scala, PySpark) to support ingestion frameworks, data processing, and platform-level libraries. Implement secure connectivity (VNET injection, Private Link, firewall, DNS, RBAC). Monitor, troubleshoot … engineering adoption. Must Have: Proven expertise with Microsoft Azure (networking, security, storage, compute). Strong proficiency in Databricks with hands-on Scala (Spark) and PySpark . Deep experience with Terraform for Azure resource deployment and governance. Hands-on with Azure DevOps pipelines (YAML, agents, service connections). Understanding ...

generative AI applications. - Perform Exploratory Data Analysis (EDA), data mining, and visualization to extract insights. - Design and implement Big Data solutions using Hadoop, Spark, PySpark, and DataLake architectures. - Develop and deploy ML models (supervised, unsupervised, tree-based, ensemble methods). - Implement deep learning architectures (CNN, RNN, LSTM) using TensorFlow … systems. Required Skills: Analytical Tools: - EDA, Data Mining, Visualization (Plotly, Matplotlib, Seaborn) - Statistical & Multivariate Analysis Big Data: - Hadoop, MapReduce, HDFS, DataBricks, Spark, PySpark - DataLake Architecture, AWS Redshift, Kinesis, EMR Machine Learning - Supervised Models: Naïve Bayes, Logistic Regression, SVM, Linear Regression, KNN - Tree-Based Models: Decision Trees, Random Forest, Gradient ...

days per week onsite What you’ll be doing: Design, build, and optimise scalable data pipelines and architecture Work hands-on with SQL, PySpark, Databricks, Azure, and Data Lake to deliver high-performance solutions Translate business needs into technical requirements and data-driven solutions Ensure best practice in data … environment What you’ll need: Proven experience as a Data Engineer or Senior Data Engineer with some team leadership responsibilities Strong technical expertise with PySpark, Databricks, Azure, and Data Lake Deep understanding of data architecture and ETL processes in cloud environments The ability to balance coding and technical delivery ...

capability to structure complex business problems and translate them into statistically sound experimental designs and solutions Proficiency in programming with experience in Python, PySpark, R, or similar languages, sufficient to implement and test statistical models and simulations, with an emphasis on correctness over advanced coding skills Demonstrated ability … testing, design of experiments, and statistical methodologies critical to experimentation science Solid programming skills in at least one language such as Python, R, or PySpark, with the ability to write clean, maintainable code to support research and implementation Experience applying statistical rigor to real-world business problems, with ...