1 to 25 of 65 Apache Spark Jobs in London

financial services organisation Your new role We are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … performance, reliability, and scalability.The role is focused on building nonperformance Apache Spark jobs, with a strong emphasis on performance optimisation. It is primarily Java-based, with Python strongly encouraged; experience across Java, Python, and Scala is ideal, with Java and Python being the preferred combination.The role sits within ...

financial services organisation Your new role We are seeking a Data Engineer to support the replacement of a legacy ETL tool with a modern Apache Spark based data platform. This is a hands-on engineering role focused on building and supporting Spark jobs, with an emphasis … Engineer. You will be responsible for development work, sprint delivery, demos, documentation, and stakeholder engagement. This position suits a mid-level engineer with strong Spark development experience rather than design, infrastructure, or management responsibilities. What you'll need to succeed Strong hands-on experience with Apache Spark ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … data Billing & revenue assurance Build models and pipelines to support network analytics, customer 360, churn prediction, fraud detection , etc. 7. Performance Optimization & Monitoring Tune Spark/Glue jobs for performance and cost. Monitor Redshift/Athena/S3 efficiency and implement best practices. Perform data quality checks and validation ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

/data platforms (Databricks, BigQuery) and building data pipelines/architectures (DBT, DLT) · Experience with batch and streaming ingestion (CDC, Kafka) and distributed processing (Spark) · Experience working with cloud infrastructure (AWS: S3, Lambda, EC2, DMS) · Experience handling structured and semi-structured data (e.g. JSON) · Familiarity with CI/ ...

Proficient SQL skills for data analysis, validation, and troubleshooting. Experience working with data warehouses, data lakes, or big data platforms(e.g., Snowflake, Redshift, BigQuery, Spark). Required Skills/Experience The ideal candidate will have the following: Experience with CI/CD pipelines and automated build and deployment processes. ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation * Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads * MongoDB - document … documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years’ experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

analytical data modeling in BigQuery using dbt . Build reliable data pipelines utilizing SQL, Python, and distributed processing frameworks (Apache Spark, Scio, Beam, or Flink). Develop clear, story-driven dashboards using tools like Looker or Tableau . Champion data quality , implement CI/CD for dbt models ...

Proven experience delivering at least one production-grade data platform Strong hands-on capability with modern data engineering tools such as Python, SQL, Airflow, Spark, Kafka, dbt, Snowflake (or equivalents) Experience with backend technologies (e.g. Java/Spring Boot) and exposure to frontend frameworks (e.g. React) Solid understanding ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

Strong SQL skills for data validation, reconciliation, and analysis. Experience working with data warehouses, data lakes, or big data platforms(e.g., Snowflake, Redshift, BigQuery, Spark). Familiarity with integrating automated tests into CI/CD pipelines(e.g., Jenkins, Azure DevOps, GitHub Actions, GitLab CI). Familiarity with version control ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

What the role entails: Some of the main duties of the Data Engineer will include: Develop high-performance data pipelines using Python (Pandas) or Spark within Palantir Foundry, ensuring seamless data transformation and integration to support analytics, reporting, and machine learning use cases across the enterprise. Design and implement ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Improve engineering standards, processes, and tooling ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Contribute to the continuous improvement ...

warehousing concepts Familiarity with version control systems, particularly Git Desirable Skills: Experience with infrastructure as code tools such as Terraform or CloudFormation Exposure to Apache Spark for distributed data processing Familiarity with workflow orchestration tools such as Airflow or AWS Step Functions Understanding of containerisation using Docker Experience ...

accommodation during the recruitment process, please let us know. Why This Role Matters Ignite, Innovate, Impact. As an AI Engineer in the AI Spark team - a focused, high‐impact product innovation team responsible for accelerating the adoption of AI‐driven capabilities across Hansen products, you will play a pivotal ...

integrations, while ensuring solutions meet regulatory standards and align with architectural best practices. Key Responsibilities: Build and optimise scalable data pipelines using Databricks and Apache Spark (PySpark). Ensure performance, scalability, and compliance. Collaborate on requirements, design, and backlog refinement. Promote engineering best practices including CI/… data pipelines that improve time-to-insight. Knowledge of secure, auditable, and compliant data workflows. Know how on optimising performance and reducing costs through Spark and Databricks tuning. Be able to create reusable, well-documented tools enabling collaboration across teams. A culture of engineering excellence driven by mentoring ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...