20 of 20 PySpark Jobs in Central London

deployment automation and infrastructure-as-code practices. What We're Looking For Essential Strong commercial experience building production-grade data pipelines using Apache Spark (PySpark preferred). Experience delivering modern data platforms using Databricks or an equivalent Spark-based platform. Strong SQL skills and experience with data modelling. Good ...

data platform on Microsoft Fabric, Databricks, Snowflake or similar, using a medallion architecture, including managing ingestion performance and forecasting platform costs. Strong Python/PySpark and SQL, with a track record of production ETL/ELT pipelines integrating data from varied business systems, including less well-documented sources. Experience ...

strong problem solving, communication, and stakeholder management skills. Requirements 4-6 years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Strong hands-on experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven experience designing and implementing metadata-driven data pipelines. ...

calculation logic (cash flows, commitments, valuations, NAV, and return metrics) across data pipelines. Automated Data Quality Frameworks: Develop automated test suites using Python (PyTest) , PySpark , and advanced SQL to catch discrepancies, nulls, and duplicate records. CI/CD Quality Gates: Integrate automated testing and data validation checks into Azure ...

best practices and driving a culture of continuous improvement. Requirements 8+ years of experience in data engineering with a strong focus on Databricks (SQL & PySpark). Extensive experience with Azure data services, including ADF, ADLS, and Azure Functions. Proven track record of designing and implementing metadata‐driven data pipelines. ...

Gen2), Purview, Cosmos DB, OpenAI, Azure ML, and AI Foundry Solid understanding of software engineering concepts, with hands‐on experience using Python, Scala or PySpark, data modelling techniques, database technologies, and SQL Confident communicator, able to explain technical concepts to non‐technical audiences and mentor junior colleagues For Senior ...

Data Lake Storage (Gen2), Purview, Cosmos DB, OpenAI, AzureML, AIFoundry, Kubernetes Understanding of software engineering tools and concepts including experience in Python, Scala or PySpark, database technologies, data modelling and SQL Confident communicator who is able to explain technical terms to non‐technical audiences and mentor junior colleagues Leads ...

Data Engineer – Microsoft Fabric & Azure London | Hybrid working | £55,000–£65,000 + 8% bonus + 7% pension + 40 days' holiday Are you a Data Engineer working with Microsoft Fabric who enjoys building solutions ...

target reconciliation, validate financial calculations, and ensure 100% data integrity across Bronze/Silver/Gold layers. Responsibilities include building automated frameworks with PyTest, PySpark, and advanced SQL, integrating checks into Azure DevOps pipelines, and monitoring schema drift and #J-18808-Ljbffr ...

Microsoft Fabric sits at the centre of the platform: Lakehouse, Warehouse, OneLake, pipelines and notebooks, built around a medallion architecture. You'll use Python, PySpark and advanced SQL to build the pipelines, models and integrations behind the business's reporting, analytics and wider data needs. You'll also: · Integrate … year in that role working with Fabric, so you've lived with your own design decisions. You'll also bring: · Strong Python/PySpark and advanced SQL · Sound data modelling, ETL/ELT, orchestration and data quality experience · A good grasp of data governance, privacy and security ...

Datatech Analytics is seeking an AWS Data Engineer to own and evolve data products in a hybrid London-based role. You will design and optimise ETL/ELT pipelines, expand the AWS data lake, and ...

Physics, Statistics, or a related quantitative field. Core Technical Stack: Hands-on proficiency with Python, SQL, and Excel, alongside exposure to platforms like Databricks, PySpark, Palantir Foundry, or R. Specialized Experience: Proven track record in delivering advanced analytics, algorithm design, operational research, or big data engineering solutions. Energy & Utilities ...

ownership of client problems, handling everything from deep data engineering knowing that may require low/no code Palantir Foundry tooling, or Python and PySpark code repos - to building user-facing front-end applications and widgets in TypeScript or REACT Lead Client Advisory: Sit directly alongside clients, goals ...

ownership of client problems, handling everything from deep data engineering knowing that may require low/no code Palantir Foundry tooling, or Python and PySpark code repos - to building user-facing front-end applications and widgets in TypeScript or REACT Client Advisory & Translation: Sit directly alongside clients, distinguish their ...

bonus and an extensive benefits package. What you'll be doing Design, build and maintain scalable data pipelines and data products using Databricks , PySpark, Spark SQL and Delta Lake. Develop and optimise ETL/ELT processes to ingest, transform and curate data from internal and external systems. Shape enterprise … help mature a growing data function. What you need to bring “Senior” level experience in data engineering, with strong hands-on Databricks skills (PySpark, Spark SQL, Delta Lake, etc) Advanced SQL and Python coding skills. A track record of building and optimising enterprise-scale data pipelines and ETL/ ...

Azure Data Engineer - 1/2 days onsite 18 month FTC Summary: Join a team building a modern Azure-based data platform. This hands-on engineering role involves designing and developing scalable, automated data pipelines ...

Job Title Senior Data Engineer Job Description Summary Senior Data Engineer | Databricks | Agentic AI | EMEA & APAC We are the team leading the data transformation of our EMEA and APAC business, and we are hiring a ...

experience with Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise in NoSQL data modelling, partitioning, indexing … experience with Pytest and automated unit/integration testing. Knowledge of Git, GitLab DevOps and CI/CD pipelines. Experience with Microsoft Fabric and PySpark data pipelines. For more information or to apply directly, please email I review applications daily and will contact shortlisted candidates as quickly as possible. ...

maintain production-quality Python and SQL code. Use Git-based source control and branch management practices across multiple deployment environments. Work with Spark and PySpark to process and analyse large-scale datasets. Develop and manage machine learning workflows within an MLOps environment. Use tooling such as MLflow, VS Code … model validation. Practical experience with Git, Agile delivery and collaborative software-development workflows. Experience using distributed data-processing and MLOps tools such as Spark, PySpark, MLflow, JupyterHub or equivalent technologies. Understanding of data-product lifecycles, data quality, governance, reproducibility and auditable insightful workflows. Ability to translate operational or business ...

Data Engineer - PySpark/AI Location: Holborn, London - hybrid, three office days per week Salary: Up to £75,000 dependent on experience Ref: J13171 Our client is a growing technology business developing data and AI products that help organisations gain deeper insight into consumer behaviour. They are expanding their … decisions Use Claude Code to support development, debugging, testing and documentation Experience required: Strong SQL skills for transforming, cleaning and loading data Commercial Python, PySpark and Pandas experience Experience with orchestration tools such as Airflow, Dagster or Celery Practical experience with Spark and EMRHands-on knowledge of AWS services ...