14 of 14 Apache Spark Jobs in Central London

you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise …/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high levels of data quality, governance and pipeline reliability. Optimise performance across large-scale distributed data processing platforms. Collaborate with ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

Strong technical project management skills with a focus on timely delivery and execution from start to finish Strong proficiency in SQL, Python, Spark, Java, Scala with the ability to write complex queries, optimise performance and data transformations with preference for experience in AI, machine learning, data-intensive platforms Proficiency ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

with Azure Cosmos DB, including query optimization, throughput management, and performance tuning. Thorough understanding of Cosmos DB Change Data Feed (CDF) and integration with Spark-based solutions. Experience working with Cosmos DB SDKs in technologies such as .NET, Java, Python, or Node.js. Strong understanding of distributed computing concepts … Apache Spark architecture. Hands-on experience developing data pipelines using PySpark. Experience with Pytest and automated testing frameworks. Strong understanding of partitioning, indexing strategies, and consistency models within Cosmos DB. Experience building scalable applications and working with concurrency patterns. Experience with Git, version control systems, and CI/ ...

datasets Preferred Qualifications Knowledge or interest in investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka ABOUT GOLDMAN SACHS At Goldman Sachs, we commit our people, capital ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

foundation models, embeddings, vector databases, RAG architectures, and agentic AI systems. Experience working with large-scale datasets using SQL, distributed processing platforms such as Spark, and cloud-based infrastructure. Strong communication and data visualisation skills, with the ability to explain complex technical concepts to technical and non-technical audiences. ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

experience in software engineering, MLOps, and deploying machine learning models at scale. Proficiency in SQL, Python, and familiarity with big data technologies (e.g., Kafka, Spark) and/or cloud platforms (e.g., GCP, AWS, or Azure). Industry knowledge: Experience in gaming or digital entertainment is a strong plus. ...

compute cluster capabilities Advanced proficiency in Python, with extensive professional experience in building production applications and services Experience of modern cluster compute - Kubernetes, Spark, Hadoop Yarn, Ray, Airflow are some of the technologies we use Experience leading work in a formal software development environment alongside other engineers ...

APIs used by researchers and engineers across the firm. Ideal candidates have deep Python expertise, hands-on experience with modern cluster compute technologies (Kubernetes, Spark, Hadoop/Yarn, Ray, Airflow) and a track record of delivering scalable infrastructure in a multi-user #J-18808-Ljbffr ...

Java, data structures and concurrency, not just framework knowledge Hands-on experience with Kafka and event-streaming frameworks such as Flink, Kafka Streams or Spark Practical Kubernetes experience, including troubleshooting production issues Experience tuning JVM performance - garbage collection, CPU hotspots and heap dump analysis Why you'll love this ...