1 to 25 of 133 Apache Spark Jobs in London

detect threats, analyze patterns, and respond to incidents in real-time. You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teams Build and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructure Design event-driven systems leveraging Kafka for high-throughput, low-latency message ...

complex legacy data pipelines into production-grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high-quality testing frameworks covering performance, data quality and reliability Leveraging Agentic AI development tools to improve … consumed We're looking for someone with experience in Staff, Principal or Lead Data Engineering roles Python and expert-level PySpark AWS Glue Apache Spark Apache Airflow Cloud-native Data Engineering architectures Large-scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing, validation ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise …/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high levels of data quality, governance and pipeline reliability. Optimise performance across large-scale distributed data processing platforms. Collaborate with ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

applications, working closely with data engineers, analysts, and platform teams. Day to day duties Developing and maintaining data pipelines and processing jobs using Apache Spark and Python Orchestrating workflows with Apache Airflow Provisioning and managing cloud infrastructure on AWS using Terraform (Infrastructure as Code) Containerising applications … teams in an Agile environment, contributing to design reviews, code reviews, and documentation Essential Skills & Experience Strong hands-on experience with Python Experience with Apache Spark for large-scale data processing Practical knowledge of AWS services (eg S3, Lambda, EC2, Glue, RDS - adjust as relevant) Experience writing ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands-on experience working with Distributed Spark based systems Experience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred [Preferred] Experience working with Apache Spark to process large-scale distributed datasets Communication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core ...

PyTorch, applying advanced proficiency in supervised and unsupervised learning techniques to solve complex image and video analysis tasks. Implement scalable data pipelines with Apache Spark and Kafka, ensuring efficient ingestion, processing, and transformation of large-scale visual datasets for model training and inference. Evaluate machine learning models using … classification performance in vision projects. Guide team members in adopting best practices for code quality, model deployment, and performance optimization, leveraging tools like Apache Airflow for workflow automation. Collaborate with internal stakeholders to define technical requirements and deliverables, ensuring alignment of vision solutions with project objectives and compliance standards. ...

production-grade solutions using BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Composer . Frameworks & Storage: Strong understanding of batch/streaming tools (e.g., Apache Spark, Apache Beam ) and when to apply different storage paradigms (relational, columnar, document, data lakes). Modern Best Practices: Active practice with ...

operating data pipelines and analytical systems using technologies such as Google BigQuery, Amazon Athena, or ClickHouse. Experience with distributed data processing frameworks such as Apache Spark and/or Apache Flink. Experience with messaging and streaming systems such as Apache Kafka or Apache Pulsar. Experience ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/ ...

considering: We are looking for an experienced AWS Databricks Engineer with strong hands-on expertise in Databricks, AWS cloud services, PySpark, Spark SQL, Delta Lake, Python, SQL, and data engineering. The candidate will be responsible for designing, developing, optimizing, and supporting scalable data pipelines and lakehouse solutions … time. Your Role: Design, develop, and maintain scalable data pipelines using Databricks on AWS. Build and optimize data processing frameworks using PySpark, Spark SQL, Python, Delta Lake, and Databricks notebooks. Develop end-to-end ETL/ELT pipelines for ingestion, transformation, validation, and consumption. Work with AWS services such ...

Risk, XVA, CCR, FRTB, portfolio risk, fixed income analytics, securitised products, liquidity risk, or managed risk-as-a-service solutions. Familiarity with Snowflake, Databricks, Spark, AWS, Azure, Confluence, Azure DevOps, Git, Tableau, Power BI, message queues, or comparable tools and platforms. Knowledge of market data vendors, data mastering, golden ...

Risk, XVA, CCR, FRTB, portfolio risk, fixed income analytics, securitised products, liquidity risk, or managed risk-as-a-service solutions. Familiarity with Snowflake, Databricks, Spark, AWS, Azure, Confluence, Azure DevOps, Git, Tableau, Power BI, message queues, or comparable tools and platforms. Knowledge of market data vendors, data mastering, golden ...

code. Modern Data & Cloud Practices : Experience with modern data engineering tooling and cloud platforms (ideally AWS and Databricks), and familiarity with distributed processing (e.g. Apache Spark). Exposure to infrastructure as code (such as Terraform and Databricks Asset Bundles) is a plus. Engineering Mindset : Understanding of testing, version ...

work closely with stakeholders and translate business requirements into effective AI solutions Desirable Generative AI experience Large Language Models (LLMs) LangChain or similar frameworks Spark, Hadoop or other big data technologies Experience with distributed computing, streaming platforms or workflow orchestration tools Experience monitoring and optimising model performance in production ...

monitoring. Strong background in CI/CD pipeline design, automation, and maintenance. Hands-on experience with data engineering tools and storage solutions (e.g., ADLS, Spark, Databricks, SQL/NoSQL databases). Experience with distributed computing and big data processing frameworks such as PySpark. Knowledge of infrastructure-as-code tools ...

improve overall operational stability of software applications and systems Designs and implements data pipelines, analytics workflows, and reporting solutions using Python, Databricks, and Spark Develops and maintains web-based applications and dashboards using modern web stacks (React, Angular, etc.) Builds and optimises data models, ETL/ELT processes ...

Experience working for a software vendor, SaaS provider, or professional services organisation. Familiarity with Ataccama or similar enterprise data platforms. Exposure to Hadoop/Spark, NoSQL, XSLT, Hive, Kafka, Snowflake, or Databricks. Python or other programming languages. Familiarity with SDLC, code management and upgrades, and Unix/Linux. Perks ...

Public Services. Experience Level Required in Years: 10 - 15 years Mandatory Required Skills Hands-on experience with Quantexa platform/Decision Intelligence Strong in Apache Spark/PySpark Proficiency in Python/Scala Experience in ETL, Data Engineering, Data Warehousing Knowledge of entity resolution, graph analytics, link analysis … ecosystem – Hive/HDFS) Detailed Job Description Design and develop solutions using Quantexa Decision Intelligence Platform Build data ingestion, transformation, and ETL pipelines using Spark/PySpark Work on entity resolution, network analytics, and graph-based data models Integrate multiple structured/unstructured data sources for analytics use cases ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

Cloud Composer and IAM You have strong Python, SQL and PySpark skills You have experience with real-time data streaming using Kafka or Spark You have a good knowledge of Data Lakes, Data Warehousing, Data Modelling You're familiar with DevOps principles, containerisation and CI/CD tools such ...

concept to production. Strong knowledge of Python and core ML libraries (e.g., spaCy, NumPy, SciPy, Transformers). Understanding of big data and cloud technologies (Spark, Hadoop, Azure, AWS, or GCP). Deep expertise in MLOps principles, scalable model deployment, and model governance. Excellent communication and stakeholder management skills with ...

experience designing and implementing multi‐agent architectures Enterprise AI Integration & Data API‐ and event‐driven integration of AI into enterprise workflows Data platforms: Databricks, Spark, Snowflake; streaming via Kafka Backend & Distributed Systems Languages: Python (expert), Java/Spring Boot (enterprise standard), Go (plus) Architecture: Microservices, domain‐driven design, event ...