15 of 15 Permanent PySpark Jobs in Central London

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations ...

security measures across the platform. Leverage Terraform or similar IaC tools for controlled and reproducible deployments. Databricks Development Develop and optimise data jobs using PySpark or Scala within Databricks. Implement the medallion architecture (bronze, silver, gold layers) and use Delta Lake for reliable data transactions. Manage cluster configurations ...

Spark Programming Ability to write Spark code for large scale data processing, including RDDs, Data Frames, and Spark SQL Python/Notebook programming PySpark programming Spark Streaming/batch processing Delta table Optimization Fabric spark jobs Java programming language, OOP knowledge Database knowledge, including Relational Database ...

overview of many different sectors. What We’re Looking For 5 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

overview of many different sectors. What We’re Looking For 10 + years, hands-on experience in AWS data engineering technologies, including Glue, PySpark, Athena, Iceberg, Databricks, Lake Formation, and other standard data engineering tools. Strong experience engineering in a front-office/capital markets environment. Previous experience ...

agentic capabilities, able to build production-grade applications not just prototypes. Strong proficiency in at least one relevant programming language such as Python or PySpark, Java, Typescript or SQL. Solid understanding of data engineering, data modelling, security and governance in enterprise settings. Experience with software engineering best practices including ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … years of experience in data engineering, with at least 3+ years of hands-on experience with Azure Databricks. Strong proficiency in Python and Spark (PySpark) or Scala. Deep understanding of data warehousing principles, data modelling techniques, and data integration patterns. Extensive experience with Azure data services, including Azure Data ...

this Senior Data Engineering role, you’ll design, develop, and maintain data platforms using technologies such as Snowflake, Databricks, Synapse/Fabric, and PySpark, ensuring the scalability, security, and performance of all data systems. As the Senior Data Engineer your role will be to include establishing and championing best ...

deliver outcomes, not experiments Helping define the standards, patterns, and ways of working as the platform evolves What you’ll bring Strong Python/PySpark and solid SQL coding experience Proven delivery of production systems at scale, not just prototypes Cloud experience and modern engineering practices, CI/ ...

/GCP (based on organization need) integrated with Databricks. Data Engineering & Platform Build Build and optimize large-scale ETL/ELT pipelines using PySpark, SQL, and Delta. Oversee data quality frameworks, metadata management, lineage, and monitoring. Leadership & Stakeholder Engagement Work closely with business leaders and product teams to translate … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

quality and structure of data Translate business needs into clear technical plans Experience required: Strong skills in data modelling SQL with either Python or PySpark Databricks or similar cloud data platform Experience communicating with both technical and non-technical users Leadership or mentoring experience Insurance or regulated sector experience ...

Title: Azure Databricks Data Engineer Primary skills: Advanced SQL, Azure Databricks, Azure Data Factory, Azure Datalake. Secondary skills: Azure SQL, PySpark, Azure Synapse. Experience: 10+ Years of Experience Hybrid - 3 Days/Week onsite is MUST About the job We are looking for an experienced Databricks Data Engineer … ingestion and transformation. • Collaborate with Azure stack modules like Data Lakes and SQL DW to handle large volumes of data. • Write SQL, Python, and PySpark code to meet data processing and transformation needs. • Understand business requirements and create data flow processes that meet them. • Develop mapping documents and transformation ...

New Permanent Opportunity for an eDV Cleared Data Engineer with AWS Cloud Experience for a leading National Security Consultancy in London. Up to £85,000 DoE plus bonuses Active eDV required London Location – full-time ...

optimise CI/CD pipelines in Azure DevOps to automate deployment of workspaces, Unity Catalog, networking, and security. Work with Databricks (Spark/Scala, PySpark) to support ingestion frameworks, data processing, and platform-level libraries. Implement secure connectivity (VNET injection, Private Link, firewall, DNS, RBAC). Monitor, troubleshoot … engineering adoption. Must Have: Proven expertise with Microsoft Azure (networking, security, storage, compute). Strong proficiency in Databricks with hands-on Scala (Spark) and PySpark . Deep experience with Terraform for Azure resource deployment and governance. Hands-on with Azure DevOps pipelines (YAML, agents, service connections). Understanding ...

days per week onsite What you’ll be doing: Design, build, and optimise scalable data pipelines and architecture Work hands-on with SQL, PySpark, Databricks, Azure, and Data Lake to deliver high-performance solutions Translate business needs into technical requirements and data-driven solutions Ensure best practice in data … environment What you’ll need: Proven experience as a Data Engineer or Senior Data Engineer with some team leadership responsibilities Strong technical expertise with PySpark, Databricks, Azure, and Data Lake Deep understanding of data architecture and ETL processes in cloud environments The ability to balance coding and technical delivery ...