1 to 25 of 43 Permanent Apache Spark Jobs in London

renowned financial services organisation Your new roleWe are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … performance, reliability, and scalability.The role is focused on building nonperformance Apache Spark jobs, with a strong emphasis on performance optimisation. It is primarily Java-based, with Python strongly encouraged; experience across Java, Python, and Scala is ideal, with Java and Python being the preferred combination.The role sits within ...

renowned financial services organisation Your new roleWe are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … Engineer. You will be responsible for development work, sprint delivery, demos, documentation, and stakeholder engagement. This position suits a mid-level engineer with strong Spark development experience rather than design, infrastructure, or management responsibilities.What you'll need to succeed Strong hands-on experience with Apache Spark - Writing ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation * Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads * MongoDB - document … documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years’ experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Improve engineering standards, processes, and tooling ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Contribute to the continuous improvement ...

strong focus on scalability, resilience, and performance. Develop and optimise scalable data pipelines , supporting both batch and streaming workloads, using technologies such as Apache Spark . Work extensively with data technologies , leveraging Python and SQL to deliver high-quality analytical and data-driven solutions. Lead the design ...

Proficiency in software engineering languages such as Python, Java, C#, JavaScript, and TypeScript.* Deep knowledge of data engineering languages and tools such as SQL, Spark, Scala, and Hadoop.* Strong background in agile and DevOps methodologies, practices, and technologies.* Proficiency with Azure DevOps, including CI/CD pipelines, automated testing ...

accommodation during the recruitment process, please let us know. Why This Role Matters Ignite, Innovate, Impact. As an AI Engineer in the AI Spark team - a focused, high‐impact product innovation team responsible for accelerating the adoption of AI‐driven capabilities across Hansen products, you will play a pivotal ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

followed Required Skills & Experience: Strong experience with Azure data platform (e.g., Data Factory, Data Lake, Synapse) Hands-on expertise with Databricks (PySpark or Spark) Solid SQL development skills Experience working with Kafka for streaming data pipelines Good understanding of lakehouse architecture, including bronze/silver/gold layering concepts ...

Solid understanding of data governance, data lineage, security practices, and regulatory compliance frameworks. Experience working with large-scale data storage and processing technologies (SQL, Spark, NoSQL, ETL/ELT tools). Excellent communication and stakeholder management skills, with the proven ability to translate complex data concepts into actionable, business ...

Docker and Kubernetes is a plus. Experience designing end to end ML data pipelines. Experience with database technologies such as PostgreSQL, MySQL, or MongoDB, Spark, Databricks and stream data processing such as Kafka, RedPanda is a plus. Direct experience in building and applying large-scale (100M+ users) machine learning ...

experience in software engineering, MLOps, and deploying machine learning models at scale.* Proficiency in SQL, Python, and familiarity with big data technologies (e.g., Kafka, Spark) and/or cloud platforms (e.g., GCP, AWS, or Azure).* Industry knowledge: Experience in gaming or digital entertainment is a strong plus. ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud ...

will have strong experience in data engineering, with advanced proficiency in SQL and Python, alongside exposure to large-scale data processing frameworks such as Apache Spark. You should have a solid understanding of data architecture, data wrangling and data quality practices, as well as experience working within financial markets ...

power, and deep knowledge of just about every industry. We believe a mix of data, analytics, automation, and responsible AI can do almost anything—spark digital metamorphoses, widen the range of what humans can do, and breathe life into smart products and services. Want to join our crew ...

influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing … Engineer or Solutions Architect certification (desirable). Databricks Certified Data Engineer or Machine Learning Associate (desirable). Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable). Knowledge of machine learning and AI on Databricks (desirable). ...