151 to 175 of 195 PySpark Jobs in London

month fixed-term contract with hybrid work (onsite 2 days per week). The role requires strong Snowflake experience, including Snowflake ML, plus PySpark and SQL skills. DBT experience is advantageous as you build and migrate data pipelines for programme use cases from Pre-Prod to Prod. You will ...

location strategy."I am the person Capital Group is looking for." Proven track record delivering solutions in complex organizations. Experience with AWS, Databricks, DBT, PySpark, React, JavaScript Terraform, etc. Fast learner, thrives in ambiguity, translates vision into product scope. Passion for automation and user experience innovation. Builds high-performing ...

DescriptionRole –Senior Databricks ArchitectTechnology – Databricks (Spark, PySpark/SQL) and data lake architecturesLocation – London, UKCompensation – Competitive (including bonus)Job DescriptionWe are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role focuses on integrating legacy … systems with cloud-native platforms to support analytics, reporting, and domain-owned data products. Primary Skills Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures. Hands-on experience with streaming technologies, especially Kafka/Confluent Kafka. Experience integrating legacy data sources such as DB2 (mainframe ...

that support critical business operations and analytics capabilities. This role requires deep hands-on expertise in Azure Databricks, Structured Streaming, Delta Live Tables (DLT), PySpark, Python, and SQL, alongside strong stakeholder management and leadership skills. The ideal candidate will have a proven track record of delivering large-scale cloud … platform performance and costs, and mentoring engineering teams to achieve successful business outcomes. Required Skills and Experience: Strong hands-on experience with Azure Databricks, PySpark, Python, and SQL Experience within financial trading, commodities trading, or energy trading environments Understanding of trading data models and trading business processes Experience building ...

modelling, Kimball, or Data Vault Exposure to modern cloud data platforms, ideally within the Microsoft ecosystem Familiarity with tools such as DBT, Python, or PySpark is beneficial Experience working in data transformation or modernisation programmes Understanding of architecture principles or a clear interest in developing into a Data Architect ...

building data models and pipelines in Snowflake, migrating programme use cases, and collaborating with Data Scientists to validate model performance. Experience with Snowflake ML, PySpark, and DBT is essential. The candidate will manage the migration from Pre-Prod to Prod, support PoC phases, and document findings during the transition ...

existing pipelines. Working within a growing data team, you'll design scalable data models, build robust ELT pipelines in Databricks using SQL, Python and PySpark, and integrate data from multiple internal and third-party systems. You'll collaborate closely with Analytics Engineers to ensure data is reliable, well-governed … architecture and help define long-term data standards, rather than simply delivering against predefined requirements. You'll need experience with: Databricks, SQL, Python and PySpark Building and optimising ELT/data pipelines Data modelling, lakehouse architecture and modern data platforms Azure data services API and third-party system integration ...

Lead the design and implementation of scalable data architectures on the Databricks Lakehouse platform. Oversee the development of efficient ETL/ELT pipelines using PySpark, SQL, and Delta Lake. Mentor and guide junior data engineers, promoting best practices in code quality, CI/CD, and performance tuning. Collaborate with … Requirements & Skills Databricks Expertise: Extensive hands-on experience with Databricks, Delta Lake, Unity Catalog, and Spark optimization. Programming Languages: Advanced proficiency in Python/PySpark and SQL. Cloud & Infrastructure: Strong background in cloud data platforms (Azure, AWS, or GCP) and cloud-native architectures. Engineering Best Practices: Experience with ...

designing, developing, and implementing enterprise-scale data solutions. The ideal candidate will possess strong expertise in Snowflake and hands‐on experience with Python, PySpark, AWS, and Airflow for building scalable, high-performance data platforms. The role demands strong technical leadership, problem‐solving capabilities, and the ability to collaborate effectively … transformationsto support large‐scale data processing Build and optimize enterprise data warehouse solutions on Snowflake. Develop data ingestion, transformation, and orchestration workflows using Python, PySpark, and Airflow. Develop and optimize complex SQL queries, stored procedures, views, streams, and tasks in Snowflake. Implement scalable cloud‐based data solutions leveraging ...

Job Title Senior Data Engineer Job Description Summary Senior Data Engineer | Databricks | Agentic AI | EMEA & APAC We are the team leading the data transformation of our EMEA and APAC business, and we are hiring a ...

engineering, or a related field Experience analysing gambling marketing activities The ability to write clear, well-structured code Hands‐on experience with Python, SQL, PySpark, or similar tools A flair for problem‐solving and storytelling with data — and the confidence to present findings to clients Sound like ...

GBMs, Elastic Net GLMs, GAMs, Decision Trees, Random Forests, Neural Nets and Clustering Experience in statistical and data science programming languages (e.g. R, Python, PySpark, SAS, SQL) A quantitative degree (Mathematics, Statistics, Engineering, Physics, Computer Science, Actuarial Science) Experience of WTW's Radar software Proficient at communicating results ...

GBMs, Elastic Net GLMs, GAMs, Decision Trees, Random Forests, Neural Nets and Clustering Experience in statistical and data science programming languages (e.g. R, Python, PySpark, SAS, SQL) A quantitative degree (Mathematics, Statistics, Engineering, Physics, Computer Science, Actuarial Science) Experience of WTW’s Radar software Proficient at communicating results ...

facing team in London. You will mentor data scientists, oversee complex data products, and ensure analytical excellence. The ideal candidate has strong SQL and PySpark experience, excellent communication skills, and a Bachelor’s degree in a related field. This position offers a collaborative environment, competitive benefits, and a chance ...

team. Nice to have: A PhD or equivalent research training in a quantitative field, and/or familiarity with Databricks, dbt, Spark/PySpark, or similar large-scale data platforms. Experience modifying infrastructure-as-code (e.g. Terraform) or contributing to production data pipelines. Experience building shared tools or platforms … them. Experience with UTXO, EVM, or other blockchain data, graph computation, and/or the cryptocurrency ecosystem. Technologies we use: Python SQL Databricks Dbt PySpark/Spark Terraform AWS Postgres AI at Chainalysis AI is not a feature at Chainalysis - it is a new way of working. One that ...

data transformationsTest framework and supporting toolingLead testing of AWS‐based Lakehouse and data pipeline solutions, including: AWS GlueApache IcebergAI‐enabled ETL acceleratorsNeo4jPython/PySpark, SQL and YAML‐driven configurationsQuality, Defect & Risk ManagementEnsure quality issues, risks issues and data defects are identified early, clearly documented and effectively triaged. Monitor, track … creation, masking and environment readiness. What experience you'll bring: Extensive experience in data engineering/test management. Strong technical capability in: Python/PySpark, SQL, Oracle, HDFS and YAMLBPSS minimum, SC not mandated but strong likelihood may be requiredStrong experience in leading test automation in data testing, ideally ...

engineering organisation behind you. Specific responsibilities include, but are not limited to: Lead the migration of legacy SQL solutions into Databricks Jobs, Workflows, and PySpark/SQL pipelines, maintaining operational SLAs throughout Define the target-state data architecture - Lakehouse, Medallion, and domain-oriented data product patterns - to support reporting … work close to commercial ambiguity. Strong, hands-on experience with Databricks, delivering enterprise-scale pipelines, workflows, and Lakehouse architectures Advanced proficiency in Python, PySpark, and SQL A proven track record migrating SQL Server workloads, stored procedures, and scheduled jobs to modern, cloud-native data platforms A solid understanding ...

Senior Databricks Architect Technology: Databricks (Spark, PySpark/SQL) and data lake architectures. Location: London, UK. Compensation: Competitive (including bonus). Job Description We are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role … integrating legacy systems with cloud-native platforms to support analytics, reporting, and domain-owned data products. Primary Skills Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures. Hands‐on experience with streaming technologies, especially Kafka/Confluent Kafka. Experience integrating legacy data sources such ...

driven outcomes. Develop interactive business applications using Foundry Workshop , Slate , and custom front-end solutions. Build robust applications and integrations using TypeScript , Python , and PySpark . Collaborate with business and technical stakeholders to translate requirements into scalable data solutions. Integrate AI/ML capabilities into data workflows and support … hands-on experience with Palantir Foundry , including: Data ingestion and integration Data transformations Foundry datasets Foundry applications and Workshop Strong programming expertise in: Python PySpark Experience building custom applications and user-facing solutions within Foundry. Strong understanding of distributed systems and microservices architecture. Experience with AWS services such ...

orchestration, experiment tracking, hyper-parameter tuning, and deploying automated model retraining pipelines. Programming & Data Engineering : Proficiency in object-oriented Python, advanced dataframes (Polars/Pyspark), and data versioning (DVC). Experience designing data storage solutions and using object-oriented SQL interfaces. Cloud & DevOps : Hands-on experience with at least … experience Advanced Education : A Master's degree or PhD in a relevant field is a strong plus. Parallelisation & Performance : Experience with parallelisation frameworks like Pyspark or Ray. Advanced Cloud & Infrastructure : Familiarity with serverless deployments (e.g., Fargate, Lambdas), infrastructure automation with Terratest or Ansible. ...

focus for this role over the next few months will include: Designing, building and maintaining production-grade data pipelines and services across AWS, PySpark, Snowflake and SQL-based transformation frameworks Developing and maintaining modular, testable and high-quality data models and transformation pipelines using DBT or equivalent structured … Infrastructure-as-Code such as Terraform Your Profile Essential skills/knowledge/experience: Experience with AWS data stack (S3, EMR, EC2, etc.) PySpark & Snowflake & CI/CD & DevOps Performance optimisation and large-scale data processing Data modelling (RDV/BDV or similar patterns preferred) Query performance tuning ...

Principal Consultant - Lead Data Analyst/Data Modeller Company: Transform Together Contract: 6-month Fixed Term Contract, with option to extend or become a permanent member of the team Compensation: OTE £90,000 Location: Hybrid ...

7Dxperts in the UK is seeking a Senior Data Engineer to design and optimize scalable data pipelines using Databricks, PySpark, Python and SQL, gaining hands-on experience with Delta Lake and lakehouse architectures. You will build ETL/ELT workflows, monitor performance, mentor juniors and contribute to cloud data ...

technical SME across Databricks, Fabric, Synapse, and Azure. Own solution design, implementation strategy, and technical delivery. Build and optimise pipelines using PySpark , Spark SQL , ADF , Synapse Pipelines , or Fabric Pipelines . Define engineering best practice, conduct code reviews, and uplift quality. Develop secure, performant SQL solutions. Manage Azure DevOps … What you’ll bring Proven leadership of Data Engineering teams. Strong commercial experience with Databricks , Microsoft Fabric , or Azure Synapse . Deep expertise in PySpark , Spark SQL , and SQL optimisation. ETL/ELT pipeline development and orchestration. Experience building scalable cloud-based data platforms. Nice to Have Snowflake experience. ...

data exploration skills Experience of working in Agile methodology Good knowledge and hands-on experience of developing Stored Procedures Experience of working with PySpark Notebooks for data ingestion and provision Practical experience of improving data quality and efficiency in data pipelines Solid understanding of DevOps and CI/… Understanding of compliance and lifecycle of data management Desirable Skills: Coding experience in any relevant language (Python, Pyspark etc.) Experience working with other Azure technologies such as Fabric, Power Automate Experience of working with non-relational databases (CosmosDB, MongoDB, etc.) and handling unstructured data in ETL pipelines Our values ...