1 to 25 of 63 Permanent Apache Spark Jobs in London

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. You'll also play a key role in pre-sales activities, solution design, and helping to drive strategic Databricks initiatives … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

domain-driven approaches CI/CD for data pipelines, version control discipline, infrastructure-as-code basics (Terraform) Streaming and big data exposure (Kafka, Kinesis, Spark, Flink) is a plus Data quality and observability tooling (Great Expectations, Monte Carlo, Soda, dbt tests) is a plus Engineering & Soft Skills: Strong software ...

consultancy or client delivery setting. Proven experience across data analysis, extraction, transformation, and loading, with hands-on knowledge of relevant technologies such as Spark, Python, Kafka, SQL, Airflow, and cloud-based ETL services. Experience designing schemas for data warehouses and data marts, assessing data quality issues, and integrating quality ...

Strong technical project management skills with a focus on timely delivery and execution from start to finish Strong proficiency in SQL, Python, Spark, Java, Scala with the ability to write complex queries, optimise performance and data transformations with preference for experience in AI, machine learning, data-intensive platforms Proficiency ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

data into something they can actually use and act on. What you'll be doing: Designing and building performant data ingestion pipelines using Apache Spark, ideally within Azure Databricks Contributing to the platform's data modelling and architecture Building dashboards and semantic models, and presenting these directly ...

work closely with stakeholders and translate business requirements into effective AI solutions Desirable Generative AI experience Large Language Models (LLMs) LangChain or similar frameworks Spark, Hadoop or other big data technologies Experience with distributed computing, streaming platforms or workflow orchestration tools Experience monitoring and optimising model performance in production ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

data models and ETL processes, to support the construction of scalable and efficient data systems. Familiarity with big data tools like DuckDB, Polars or Apache Spark for processing out-of-memory datasets" . Proven experience collaborating with first-team stakeholders within a professional sports environment, demonstrating strong interpersonal ...

Oracle). Comprehensive knowledge of European and international gas and power markets, covering physical and financial trading, forward/futures/options, basis risk, spark and dark spreads, gas balancing, power scheduling, nominations, LNG and cross-commodity optimisation. Proven track record of defining and delivering enterprise-level architecture ...

with the ability to deliver incremental data modelling solutions in collaboration with engineering teams. Technical Environment (Desirable) Microsoft Fabric Azure Data Factory Python PySpark Apache Spark Desirable Skills Azure DevOps Contract Details Sector: Banking/Financial Services Location: London – Office based 4 days per week Contract Type ...

Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial ...

backend roles Strong Python; experience with distributed data systems Relational DB experience (PostgreSQL preferred) NoSQLExposure to vector databases (Qdrant, Milvus, pg vector) Bonus: Spark/Airflow/Kafka, cloud (GCP/AWS/Azure), AI/ML platform experience, early-stage startup bac kground Why Join Build the data ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake). Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure) is a plus Why join High impact and ownership ...

scale datasets Preferred Qualifications Knowledge or interestin investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka Goldman Sachs Highlights At Goldman Sachs, we commit our people, capital ...

datasets Preferred Qualifications Knowledge or interest in investment banking or financial instruments Experience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue) Experience with near real time and event-based systems like Kafka ABOUT GOLDMAN SACHS At Goldman Sachs, we commit our people, capital ...

Experience designing and enforcing data contracts and quality SLAs in distributed data ecosystems. Proficiency in Python or Java, and experience building pipelines with Airflow, Spark, or Kafka. Strong understanding of data governance principles, privacy regulations (such as the GDPR, CCPA), and secure data handling practices. Familiarity with metadata management ...

auction systems Excellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams. Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructure Experience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones. What ...

isolation, Kubernetes/container orchestration, serverless patterns. Working knowledge of cloud data platforms (Snowflake, Databricks, BigQuery, Azure Synapse) and real-time streaming (Kafka, Flink, Spark Streaming). Familiarity with generative AI patterns (LLM fine-tuning, RAG, prompt engineering, function calling, agent frameworks) and graph/knowledge graph technologies. DevOps ...

such as MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management. Strong Python experience, with helpful knowledge of Spark or PySpark for working with large‐scale data and machine learning systems. An understanding of feature stores and related data technologies used to support ...

Company Description Publicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom, and Zenith, as well as global accelerator PMX; immersive experience group PMCI and access to integrated platform‐based technologies and offerings from CJ, Epsilon ...

ownership.Experience designing and enforcing data contracts and quality SLAs in distributed data ecosystems.Proficiency in Python, Java, or Scala, and experience building pipelines with Databricks, Spark, or Kafka.Strong understanding of data governance principles, privacy regulations (GDPR, CCPA), and secure data handling practices.Familiarity with metadata management and data catalog tools (e.g.DataHub ...