1 to 25 of 59 PySpark Jobs in London

modern UI patterns, develop framework is essential Back-end: Expertise in C# .NET Core, REST APIs, and microservices-based development is essential Python (pyspark): Experience with data pipelines and analytics is a plus. Power BI: Familiarity with developing dashboards and data models. Architecture: Knowledge of microservices, API design ...

Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud Storage ). Exposure ...

within financial services environments You have expertise with GCP including BigQuery, Pub/Sub, Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data ...

Azure ML. Key skills and responsibilities, Demonstrated hands-on experience with Microsoft Fabric and Power BI. Extensive knowledge in data modelling, DAX, Python/PySpark, SQL/KQL. Proven record of technical leadership and effective stakeholder engagement. Design and deliver comprehensive Fabric solutions, including OneLake, Lakehouse, Warehouses, and Power ...

Azure ML. Key skills and responsibilities, Demonstrated hands-on experience with Microsoft Fabric and Power BI. Extensive knowledge in data modelling, DAX, Python/PySpark, SQL/KQL. Proven record of technical leadership and effective stakeholder engagement. Design and deliver comprehensive Fabric solutions, including OneLake, Lakehouse, Warehouses, and Power ...

with CDPs that are SQL-native. Treasure Data is a significant advantage; exposure to mParticle, Tealium, Segment or Salesforce CDP will translate. Production-grade PySpark/Databricks experience — building scalable data pipelines, optimising performance, and operating Delta Lake/Lakehouse architectures in a real production environment. Solid DevOps craft ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data ...

agile environment What we need Strong experience as a Data Engineer in Azure environments Strong hands-on experience with: Azure Data Factory Databricks/PySpark SQL Server/T-SQL Experience building production-grade data pipelines and data models Exposure to identity/access/security data Understanding ...

Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

Lloyd's of London or London Market insurance sector Deep understanding of Delta Lake, Medallion architecture, Lakehouse patterns and data modelling. Strong Python, PySpark and Spark SQL capability Solid understanding of data governance Proven track record of ownership, delivery and operating effectively in fast-paced environments. What ...

exposure to enterprise-level data platforms and best practices. The role will involve developing and maintaining data pipelines, transforming data using Python, SQL and PySpark , integrating data from various internal and external sources, and supporting day-to-day data operations. You’ll also contribute to data quality initiatives, documentation ...

/Ansible Work closely with engineering teams to deliver real-world ML systems across multiple sectors. Tech Stack You’ll Work With Python, SQL, PySpark PyTorch, TensorFlow, Scikit‐learn MLflow (core focus), Databricks, Azure IaC: Terraform, ARM, Ansible Git, Azure DevOps, Jenkins What We’re Looking For ✔️ 3+ years ...

data modelling for Finance and Enterprise data products, working closely with architecture where required. Implement and maintain data pipelines and ETL workflows in Databricks (PySpark, Delta Lake). Contribute to CI/CD pipelines for data applications using Azure DevOps and infrastructure-as-code (Terraform) in line with established … influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing ...

ensuring solutions meet regulatory standards and align with architectural best practices. Key Responsibilities: Build and optimise scalable data pipelines using Databricks and Apache Spark (PySpark). Ensure performance, scalability, and compliance (GxP and other standards). Collaborate on requirements, design, and backlog refinement. Promote engineering best practices including ...

Responsibilities Build and maintain Databricks Delta Live Tables (DLT) pipelines across Bronze → Silver → Gold layers, ensuring quality, scalability, and reliability. Develop and optimise Spark (PySpark) jobs for large-scale distributed processing. Design and implement streaming data pipelines with Kafka/MSK, applying best practices for late event handling ...

financial modelling Knowledge graph experience Probabilistic programming experience (e.g. PyMC, Stan). Causal Inference knowledge/experience. Our technology stack Python and Jupyter Notebooks. PySpark for processing big data. Databricks: Workflows, Asset Bundles, MLFlow. About Us Who we are: PEI Group is a subscriber‐focused business intelligence company. With ...

RBAC and governance controls across Azure services Tech stack Azure Azure Databricks Terraform Azure DevOps/CI/CD Python SQL Apache Spark/PySpark Entra ID/RBAC/governance Networking/platform security/automation What were looking for Strong experience in Azure platform engineering Strong hands ...

Working closely with data engineering and platform teams • Translating complex outputs into clear, actionable insight • Agile, cross-functional delivery Tech Python, SQL, R, Databricks, PySpark, GIS tools, optimisation frameworks, cloud platforms Requirements Degree or higher in Data Science, Mathematics, Operational Research, Engineering, Physics, Geography, or similar Experience working with ...

Doing Designing and delivering end‐to‐end data and analytics solutions across complex enterprise environments Building data pipelines and workflows using Python, SQL, PySpark, and low‐/no‐code tooling Supporting the development of user‐facing applications and dashboards (TypeScript/React exposure beneficial) Working closely with stakeholders ...

Microsoft Data & Analytics within consultancy or large enterprise environments Strong expertise in Microsoft Fabric, Purview, Power BI , data architecture and modelling Experience with Python, PySpark and T-SQL Confident influencing senior stakeholders and shaping enterprise solutions Eligible for SC Clearance (five years continuous UK residency) Please apply with ...

scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical leadership … Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub Actions ...

design, infrastructure, or management responsibilities. What you'll need to succeed Strong hands-on experience with Apache Spark - Writing and tuning Spark jobs/PySpark development experience. Experienced with Airflow and SQL. Strong experience working in with containerised environments using Kubernetes. Experience with programming in Python or Scala Experience ...

outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

covered) Salary: £45,000 – £95,000 - dependent on level/experience (mid, senior, principle) Security Clearance: SC active Requirements: Databricks and Spark/PySpark (absolute must haves) Azure and AWS experience Data warehouses/lakehouse platforms CI/CD and modern engineering practices Public sector/Central government ...

landscapes and building workflows that support better decision making. The role is end to end. That means data engineering in Palantir Foundry, Python and PySpark, through to user-facing applications in TypeScript or React, and embedding AI workflows using AIP Logic and AIP Analysts. You will have ownership over ...