1 to 25 of 52 PySpark Jobs in London

ingestion and transformation pipelines from multiple source systems. Strong SQL skills and experience working with relational data models. Experience with Python and/or PySpark for data transformation and automation. Understanding of Power BI, semantic models, DAX and reporting requirements. Experience working with APIs, JSON, CSV, Parquet, Delta tables ...

projects. Hands-on AWS experience. Experience working within Azure environments. Strong SQL skills and understanding of modern data engineering principles. Desirable Skills Python and PySpark development. AWS services including EMR, S3, Lambda, Athena, and Redshift. Apache Spark and large-scale batch processing. Terraform, CI/CD pipelines, and Infrastructure ...

data engineering Strong programming expertise in Python Hands-on experience with Databricks (workflows, notebooks, jobs, Delta Lake) Strong experience with Apache Spark/PySpark Experience building scalable ETL pipelines Proficiency in SQL and relational databases Experience with cloud platforms (AWS/Azure/GCP) Experience with modern data warehousing ...

OneLake, Data Factory, Synapse Data Engineering/Data Science, and Real-Time Analytics). Core Data Stack: High proficiency in advanced SQL , Python/PySpark, and data lakehouse architectures (Delta Lake). Cloud Infrastructure: Strong understanding of Microsoft Azure data services (Azure Data Factory, Azure Synapse, Azure ...

Duration: 5 months Essential Skills: Informatica Data project experience AWS Azure Desirable Skills: Hands-on expertise in AWS & Spark: Amazon EMR, S3, Lambda; strong PySpark/Python and SQL for large-scale batch processing. Data engineering at scale in government or similarly complex domains, including performance tuning and data ...

working with flexibility and autonomy Key Experience Deep end-to-end experience delivering enterprise Databricks analytics solutions Expertise in Lakehouse Architecture, Delta Lake, Spark, PySpark Strong data modelling skills (3NF, Kimball, Data Vault) Experience exposing Delta Lakehouse to Power BI or Tableau Skilled in Python, Scala ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

+ environment configurations. Analysing current Data Engineering solutions + identifying data migration requirements. Onboarding datasets, users + workloads onto target platform. Data Engineering: Python, PySpark, SQL, Data Pipeline Development, ETL/ELT Design, Data Modelling, Data Quality, Data Validation. AWS Cloud Platform, S3, IAM, CI/CD concepts, DevOps ...

analytical use cases Ability to work closely with both technical and non-technical stakeholders, translating business requirements into data solutions Experience with Python or PySpark, particularly within Fabric Notebooks Exposure to Azure data services such as Data Factory, Synapse or Data Lake Power BI experience, including supporting reporting ...

essential - More to React JS and Java script Back-end: Expertise in C# .NET Core, REST APIs, and microservices-based development is essential Python (pyspark): Experience with data pipelines and analytics is a plus. Power BI: Familiarity with developing dashboards and data models. Architecture: Knowledge of microservices, API design ...

enterprise-scale analytics environments Technical Skills Advanced SQL/T-SQL Python for data engineering, automation, and transformation DAX for Power BI semantic modelling PySpark/Spark (highly desirable) Microsoft Azure ecosystem knowledge Our client is looking for someone who combines strong technical depth with excellent communication and stakeholder ...

level Performance tuning of complex SQL/T-SQL at scale Confidence taking ownership of workstreams and mentoring junior team members Desirable for both: PySpark/Python in Fabric Notebooks, Azure Data Factory/Synapse/Data Lake, Power BI, AI/ML exposure ...

ADLS).· Solid understanding of Scrum practices and a strong Agile mindset.· Advanced proficiency in Python and its key data and ML libraries (NumPy, PySpark, Scikit learn, TensorFlow/PyTorch)· Working knowledge of generative AI models (e.g., ChatGPT, Claude), Databricks GenAI tooling (including embedding models), and modern data structures ...

with Databricks SQL endpoints for data access. Ensure performance, scalability, and security of backend systems. Databricks & Data Engineering Develop ETL pipelines and workflows using PySpark on Databricks. Work with Delta Lake, Unity Catalog, and Databricks REST APIs. Optimize queries and pipelines for cost efficiency in cloud environments (Azure/ ...

years consulting (or equivalent) experience required 7+ years Databricks solution design, implementation, engineering experience required Either 5+ years building data pipelines in Databricks using PySpark, Scala and/or Spark SQL and ability to work across structured, semi-structured and unstructured data OR 5+ years ML Engineering ...

experience. Knowledge Essential Good working knowledge with integration and transformation within Cloud Data Warehouses (e.g Azure, AWS, GCP) including languages such as Python, SQL, PySpark, Java Knowledge and experience with programming languages such as React, Typescript, Javascript is desirable Skills and Experience Essential Can deliver well-scoped work independently ...

plateformes de données (batch & temps réel) au sein d’environnements distribués basés sur Cloudera. Vos missions : Développement & Industrialisation Développer des pipelines de traitement en PySpark . Mettre en place des flux temps réel via Kafka (topics, partitions, schémas, offsets). Modéliser et optimiser les schémas Cassandra (tables, clés, clustering ...

programming, concurrency, multithreading) Expertise in RESTful API design, OpenAPI/Swagger documentation and API lifecycle best practices Hands-on experience with ML libraries: PyTorch, PySpark, scikit-learn, Hugging Face Transformers Proficiency in AI tooling: Azure ML, Databricks, MLflow, LangChain, LangGraph Proven experience with Git, unit testing, containerisation and agile ...

engineers on new capabilities Opportunity to gain exposure to data modelling and machine learning projects YOUR SKILLS AND EXPERIENCE: Experience with Python and PySpark (commercial, academic, or project-based) Exposure to Databricks or similar data platforms Strong desire to learn and develop within a collaborative team Understanding of data ...

Looking For: Proven experience working with Palantir Foundry in production environments Strong engineering background with expertise in at least one of: Python, SQL, PySpark, or TypeScript Strong understanding of modern data architectures & distributed systems Ability to work in fast-paced, ambiguous environments Excellent stakeholder engagement and communication skills Experience ...

DataBricks, Snowflake or MS Fabric. Comfortable engaging directly with stakeholders, taking ownership of technical decisions, and contributing to the broader data strategy. Essential SQL PySpark/Python DBT Azure knowledge (i.e. Logic apps) Power BI/Fabric Semantic Models Ability to work with stakeholders with their own operational pressures. ...

DataBricks, Snowflake or MS Fabric. Comfortable engaging directly with stakeholders, taking ownership of technical decisions, and contributing to the broader data strategy. Essential SQL PySpark/Python DBT Azure knowledge (i.e. Logic apps) Power BI/Fabric Semantic Models Ability to work with stakeholders with their own operational pressures. ...

/or algorithms Our technology stack 💻 Python and associated ML/DS libraries (scikit-learn, numpy, LightlGBM, Pandas, LangChain/LangGraph, TensorFlow, etc..) PySpark AWS cloud infrastructure: EMR, ECS, Athena, etc. MLOps: Terraform, Docker, Airflow, MLFlow More information: Enjoy fantastic perks like private healthcare & dental insurance, a generous work ...

travel to Leeds/Newcastle) Clearance: Active SC Clearance required Rate: Up to £425/day (Inside IR35) Tech Stack: AWS (EMR, S3, Lambda), PySpark, Python, SQL, CI/CD Focus: Building scalable pipelines, modernising ETL, supporting MI/Power BI Level: Mid to Senior - mentoring & architectural input expected ...

tools, and emerging technologies. \* Develop reference architectures, solution blueprints, and best-practice playbooks for Databricks implementations. \* Architect scalable ETL/ELT pipelines using PySpark, Spark SQL, Delta Live Tables, and Databricks Workflows. \* Implement performance optimization strategies for notebooks, clusters, job orchestration, and data storage. \* Guide teams in building curated ...