26 to 50 of 60 Apache Spark Jobs in the City of London

Join a leading multi-strategy hedge fund in London, where we leverage cutting-edge technology and data to drive investment decisions across diverse asset classes. We're looking for a skilled Data Engineer to play a crucial role in building More ❯

Senior Data Engineer/Data Analytics Engineer - Full Remote Working from anywhere in the UK SQL+Python+ETL+ Apache Airflow Our client is a global leading and fast growing high tech company: -Over 6,500 employees across 20+ offices; -300 Million+ active users on some of the platforms they developed; -Cutting edge AR and VR technologies, 3D printing, etc. They are … FinTech team! You will develop scalable data pipelines, ensure data quality, and support business decision-making with high-quality datasets. -Work across technology stack: SQL, Python, ETL, Big Query, Spark, Hadoop, Git, Apache Airflow, Data Architecture, Data Warehousing -Design and develop scalable ETL pipelines to automate data processes and optimize delivery -Implement and manage data warehousing solutions, ensuring … data integrity through rigorous testing and validation -Lead, plan and execute workflow migration and data orchestration using Apache Airflow -Focus on data engineering and data analytics Requirements: -5+ years of experience in SQL -5+ years of development in Python -MUST have strong experience in Apache Airflow -Experience with ETL tools, data architecture, and data warehousing solutions More ❯

in Python with libraries like TensorFlow, PyTorch, or Scikit-learn for ML, and Pandas, PySpark, or similar for data processing. Experience designing and orchestrating data pipelines with tools like Apache Airflow, Spark, or Kafka. Strong understanding of SQL, NoSQL, and data modeling. Familiarity with cloud platforms (AWS, Azure, GCP) for deploying ML and data solutions. Knowledge of MLOps More ❯

curious, keep learning, and help shape our digital platforms 🧠 What we’re looking for Proven experience as a Data Engineer in cloud environments (Azure ideal) Proficiency in Python, SQL, Spark, Databricks Familiarity with Hadoop, NoSQL, Delta Lake Bonus: Azure Functions, Logic Apps, Django, CI/CD tools 💼 What you’ll get from Mars A competitive salary & bonus Hybrid working More ❯

delivery Core Experience: Proven track record in data architecture , either from a delivery or enterprise strategy perspective Deep experience with cloud platforms (Azure, AWS, GCP) and modern data ecosystems (Spark, Databricks, Kafka, Snowflake) Strong understanding of Data Mesh , Data Fabric , and data product-led approaches Data modelling expertise (relational, dimensional) and familiarity with tools like Erwin , Sparx , Archi Experience More ❯

Key Responsibilities Design and implement real-time data pipelines using tools like Apache Kafka, Apache Flink, or Spark Streaming. Develop and maintain event schemas using Avro, Protobuf, or JSON Schema. Collaborate with backend teams to integrate event-driven microservices. Ensure data quality, lineage, and observability across streaming systems. Optimize performance and scalability of streaming applications. Implement CI … development. Strong programming skills in Python, Java, or Scala. Hands-on experience with Kafka, Kinesis, or similar messaging systems. Familiarity with stream processing frameworks like Flink, Kafka Streams, or Spark Structured Streaming. Solid understanding of event-driven design patterns (e.g., event sourcing, CQRS). Experience with cloud platforms (AWS, GCP, or Azure) and infrastructure-as-code tools. Knowledge of More ❯

between the data warehouse and other systems. Create deployable data pipelines that are tested and robust using a variety of technologies and techniques depending on the available technologies (Nifi, Spark) Build analytics tools that utilise the data pipeline to provide actionable insights into client requirements, operational efficiency, and other key business performance metrics. Complete onsite client visits and provide More ❯

Role: Backend Python & AI Engineer Location: London | Hybrid Working (2 days in Office) An ambitious and fast-growing financial technology company on a mission to transform how people understand and interact with their personal finances. By combining intelligent technology with More ❯

code Experience working on distributed systems Strong knowledge of Kubernetes and Kafka Experience with Git, and Deployment Pipelines Having worked with at least one of the following stacks: Hadoop, Apache Spark, Presto AWS Redshift, Azure Synapse or Google BigQuery Experience profiling performance issues in database systems Ability to learn and/or adapt quickly to complex issues Happy More ❯

Partner with cross-functional teams to deliver robust data solutions 💡 What You’ll Bring Strong hands-on experience building streaming data platforms Deep understanding of tools like Kafka , Flink , Spark Streaming , etc. Proficiency in Python , Java , or Scala Cloud experience with AWS , GCP , or Azure Familiarity with orchestration tools like Airflow , Kubernetes Collaborative, solutions-focused mindset and a willingness More ❯

cost-optimal. Strong experience in different software testing paradigms to ensure consistency and correctness of our data. Bonus Skills Knowledge of real-time or stream processing systems (e.g. Kafka, Spark Streaming). Domain experience in energy, IoT, or working with unreliable/messy datasets. Frontend awareness—able to contribute to or reason about how data is visualized and interacted More ❯

science solutions in a commercial setting. MSc in Computer Science, Machine Learning, or a related field. Experience building data pipelines (realtime or batch) & data quality using modern toolchain (e.g., Apache Spark, Kafka, Airflow, dbt). Strong foundational knowledge of machine learning and deep learning algorithms, including deep neural networks, supervised/unsupervised learning, predictive analysis, and forecasting. Expert More ❯

Strong track record delivering production-grade ML models Solid grasp of MLOps best practices Confident speaking to technical and non-technical stakeholders 🛠️ Tech you’ll be using: Python, SQL, Spark, R MLflow, vector databases GitHub/GitLab/Azure DevOps Jira, Confluence 🎓 Bonus points for: MSc/PhD in ML or AI Databricks ML Engineer (Professional) certified More ❯

improvement across the team 🧰 What You’ll Need Strong experience leading data engineering teams in high-growth environments Deep expertise with real-time data processing tools (e.g. Kafka, Flink, Spark Streaming) Solid hands-on knowledge of cloud platforms (AWS, GCP or Azure) Strong proficiency in languages like Python, Java or Scala Familiarity with orchestration tools such as Airflow or More ❯

Looker. • Knowledge of CI/CD processes and infrastructure-as-code. • Eligible for SC clearance (active clearance highly advantageous). Desirable Skills • Exposure to large data processing frameworks (e.g., Spark, Hadoop). • Experience deploying data via APIs and microservices. • AWS certifications (Solution Architect Associate, Data Analytics Speciality, etc.). • Experience in public sector programmes or government frameworks. Package & Benefits More ❯

Principal Data Engineer, Consulting London Based You must be eligible for SC Clearance Role Overview The Principal Data Engineer will be responsible for designing and implementing cloud-based data solutions using a range of AWS services. This role involves working More ❯

Our client is a leading UK-based consultancy seeking a skilled professional to shape data strategies, mentor dynamic teams, and deliver cutting-edge solutions. With hands-on expertise in Spark, SQL, and cloud platforms like Azure, you’ll lead end-to-end projects, drive innovation, and collaborate with clients across industries. What You’ll Do: Lead complex data engineering … multi-project environments experience. Expertise in ETL, data modelling, and Azure Data Services. Experience in designing and implementing data pipelines, data lakes, and data warehouses. Hands-on experience with Apache Spark and bonus points for Microsoft Fabric Any certifications are a bonus. Benefits: Competitive base salary Hybrid work once a week into their Central London office 25 days More ❯

Familiar with observability tools, logging frameworks, and performance monitoring Desirable Skills: Background in serverless technologies (e.g. Lambda, Step Functions, API Gateway) Experience with data tools like EMR, Glue, or Apache Spark Understanding of event-driven architecture (EventBridge, SNS, SQS) Knowledge of AWS database offerings including DynamoDB and RDS Familiarity with multi-region deployments and failover strategies AWS certifications More ❯

or AI, including leadership roles. Deep expertise in machine learning, NLP, and predictive modelling. Proficient in Python or R, cloud platforms (AWS, GCP, Azure), and big data tools (e.g. Spark). Strong business acumen, communication skills, and stakeholder engagement. If this role looks of interest, please apply here. Please note - this role cannot offer visa sponsorship. More ❯

also a plus. Proficient in statistical modelling and ML techniques, including regression, Bayesian inference, and probabilistic programming. Experience with PyMC, MCMC methods, and scenario-based optimization. Strong SQL (or Spark) skills for large-scale data extraction and preparation. Solid foundation in statistical theory and applied mathematics. Experience working in cloud-based environments (preferably Azure). Advanced degree (MSc/ More ❯

concepts in ML, data science and MLOps. Nice-to-Have : Built agentic workflows/LLM tool-use. Experience with MLFlow, WandB, LangFuse, or other MLOps tools. Experience with AirFlow, Spark, Kafka or similar. Why Plexe? Hard problems: we're automating the entire ML/AI lifecycle from data engineering to insights. High ownership: first 5 engineers write the culture More ❯

systems Experience with cloud platforms (GCP, AWS or Azure), especially managed ML services like SageMaker or Vertex AI Proficiency in SQL and working knowledge of distributed computing tools like Spark or Dask Nice to Have: Experience with natural language processing (NLP) e.g., using LLMs, transformers, text classification Familiarity with Graph ML (e.g., for customer network analysis or fraud detection More ❯

code Desired Skills (Bonus Points): Proven experience in recommender systems, behavioural AI, and/or reinforcement learning. Building data pipelines (realtime or batch) & data quality using modern toolchain (e.g., Apache Spark, Kafka, Airflow, dbt). PhD in Computer Science, Machine Learning, or a closely related field What We Offer: Opportunity to build technology that will transform millions of More ❯

on cutting-edge blockchain data infrastructure in a highly dynamic environment. What We're Looking For: 5+ years of engineering experience Strong expertise in data engineering: – Proficient in Python , Spark – Advanced SQL skills Proven experience with data ingestion pipelines Hands-on with cloud platforms – ideally AWS Solid experience working with Big Data and time series databases You’ll be More ❯

real-time data systems. Strong track record in networks, telecom, or customer experience domains (preferred). Proficiency in cloud platforms like GCP, AWS, or Azure; plus tools like Kafka, Spark, Snowflake, Databricks. Skilled collaborator with excellent stakeholder and vendor management capabilities. Confident communicator with the ability to bridge technical and business audiences. London – Hybrid Circa £550pd Outside IR35 More ❯