126 to 142 of 142 PySpark Jobs in London

Experience: 610+ Years Key Skills: MDM, Databricks, PySpark, SQL, Delta Lake, Unity Catalog, Data Quality, Data Governance Role Summary We are looking for an experienced MDM/Databricks Data Engineer to build scalable master data and integration solutions on a modern Databricks Lakehouse. The role will focus on common … entity matching, deduplication, and survivorship rules. Integrate data from multiple source systems into the Databricks Lakehouse . Develop and optimize data pipelines using Databricks, PySpark, SQL, and Delta Lake . Build and enhance Bronze, Silver, and Gold data layers. Implement data quality, standardization, reconciliation, and validation processes. Support Unity ...

helping to raise the bar on engineering practice. The foundations still matter: this is a data engineering role at heart, built on Python, PySpark, and platforms like Databricks, Snowflake, or AWS. What you'll be doing: Take ownership of designing, building, and maintaining the production data and feature pipelines … that catches drift and degradation before a client does Build and maintain the batch and ELT pipelines the ML work sits on, using Python, PySpark, and cloud data services (e.g., AWS Glue, EMR, S3, Redshift, Athena, or the Databricks/Snowflake equivalents), making sound decisions on architecture, data modelling ...

Test Automation Engineer - SC Cleared £300 per day - Outside IR35 6-month contract with scope to extend Central London - 3x days a week We're seeking a Test Automation Engineer to automate testing for data ...

organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements: Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong … Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously in a fast-moving environment Experience with Databricks Apps, Lakebase, Airflow, Azure Data Factory, monitoring platforms or financial services would be beneficial. ...

forecasting, gradient boosting, clustering, NLP and Bayesian methods Strong MLOps: orchestration, experiment tracking, hyperparameter tuning and automated retraining pipelines Object-oriented Python, Polars or PySpark, and data versioning (e.g. DVC), plus experience designing data storage Hands-on experience with at least two of AWS, Azure and GCP, including deployment … solid grasp of CI/CD, testing and containerisation Nice to have A Masters or PhD Experience with parallel processing frameworks such as PySpark or Ray Serverless deployments (e.g. Fargate, Lambda) and infrastructure automation with tools like Terratest or Ansible Whats on offer Salary: Up to £113,000 Hybrid ...

sets Programming experience (focus on machine learning): SQL and Python’s Data Science stack; good knowledge of at least one big data framework (e.g., PySpark, Hive, Hadoop) is a plus. R, SPSS, and SAS are considered nice‐to‐have Strong understanding of machine learning methods and experience applying them … into production or applied business environments is a plus While we advocate using the right tech for the right task, we often leverage: Python, PySpark, the PyData stack, SQL, Airflow, Databricks, Kedro (our open‐source data pipelining framework), Dask/RAPIDS, Docker, Kubernetes, and cloud solutions such ...

role: Lead Developer role for a global organization, responsible for designing, implementing and optimizing Python and cloud based solutions using AWS and PySpark for data intensive and regulatory focused applications. The role requires strong experience in PL SQL and AWS services, owning end to end technical delivery, code quality … solutions on AWS cloud that leverage core services to support highly available data processing pipelines and transaction workflows aligned with enterprise standards. Implement optimized PySpark jobs for large scale data transformations, ensuring efficient resource usage, predictable performance and alignment with agreed data models and quality rules. Create and refine ...

KDR Talent Solutions Ltd. in Greater London is hiring a Senior Data Engineer on a hybrid, permanent basis with a salary of £80,000–£85,000 plus a generous bonus. You will be hands-on ...

working as a Data Analyst, ideally within FinTech or Financial Services Exposure to Derivatives or other Financial instruments Strong SQL experience Strong Python experience (PySpark, Pandas, Jupyter Notebooks etc.) Airflow/Algo for workflow management Git Any exposure to compressed file formats, such as Parquet or HDF5, is highly … Bash skills are highly desirable Job title: Data Analyst Client: Elite FinTech Salary: £65,000-£100,000 + Bonus Location: London Skills: SQL, Python, PySpark, Airflow, Linux #J-18808-Ljbffr ...

this area will be required to support the Data Scientists to run a Proof of Concept (PoC) Migration Phase. Support in transforming Programme PySpark for the respective use cases, as part of migration preparation. Work collaboratively with nominated Data Scientists (Programme) to validate the functionality/performance … important the resources work collaboratively with the nominated Data Scientists to help with model/data pipeline build standardisation, issue resolution, etc. PySpark experience. The pipeline is written in PySpark, it is vital that the engineers can quickly get up to speed on the pipeline, advise on efficiency ...

Role: Senior Developer - Python, AWS & PySpark Location: London (Hybrid) About the role We are looking for a Senior Developer to join a global organization and take ownership of end-to-end technical delivery. You will design, implement and optimize Python and cloud-based solutions using AWS and PySpark. These … term reuse and evolution. Develop secure, efficient AWS solutions that use core services to support highly available data pipelines and transaction workflows. Implement optimized PySpark jobs for large-scale data transformations, with efficient resource usage and predictable performance. Create and refine PL/SQL procedures and queries that protect ...

development projects. The ideal candidate will have a strong background in ETL, data validation, and SQL, coupled with expertise in test automation engineering using PySpark, Python, and Databricks. You will play a critical role in ensuring the delivery of high-quality software solutions, employing AI-assisted testing and validation … maintain test plans and test cases for ETL processes. Conduct thorough data validation and verification using SQL. Develop and execute automated test scripts using PySpark, Python, and Databricks. Integrate AI-assisted testing methodologies to improve testing efficiency and accuracy. Collaborate with cross-functional teams to identify and resolve defects. ...

AWS Data EngineerLondon | Hybrid | Competitive salary, DOE | Fast interview processJoin an established market leading business with a start-up environment that is building a unique data platform with AI at the core. An innovative consumer ...

Developer to own end-to-end technical delivery in London. You will design and build scalable Python applications and secure AWS data pipelines, leveraging PySpark for large-scale transformations. Work closely with product owners to translate regulatory requirements into robust technical solutions, review code to uphold standards, and partner ...

data products — from client scoping and solution design to execution and ongoing delivery. Ensure high analytical and technical standards across the team, including SQL, PySpark, Python, and data validation best practices. Act as an escalation point for complex analytical challenges, providing hands‐on guidance where needed. Partner closely with … perfect job for someone who has: Proven experience managing and mentoring data analysts or analytics teams. Strong hands‐on experience with SQL and PySpark for data analysis — Must. Strong Python scripting experience — Must. Excellent communication skills in English, with the ability to translate complex analysis into clear business insights. ...

. The role focuses on building data models and pipelines in Snowflake, including Snowflake ML, and collaborating with Data Scientists to migrate and validate PySpark-based use cases during the September–December window. Ideal candidates will bring Snowflake expertise, PySpark experience, and DBT knowledge. #J-18808-Ljbffr ...

Engineers/data science backgrounds with transaction monitoring background The role combines transaction monitoring domain expertise with technical skills in Data Science, Machine Learning, PySpark programming. Experience in designing and deploying AI/ML products, hands-on and as a manager Develop and maintain AI driven monitoring engines that … Machine Learning models on Transaction Monitoring systems. Significant experience as a Data Scientist for complex financial services environments. Technical skills in AI/ML, PySpark, Data Science and knowledge on banking regulatory compliances and financial crime domain knowledge. ...