1 to 25 of 53 Apache Spark Jobs in London

/ELT) to support analytics and downstream use cases Developing scalable data solutions using Python , with a focus on maintainability and performance Working with Apache Spark/PySpark to process and transform large datasets Supporting the ingestion, transformation and validation of complex financial data Improving the performance, reliability … That Will Help You Succeed Essential Commercial experience as a Data Engineer at a mid‐level Strong Python development skills Hands‐on experience with Apache Spark/PySpark Solid experience building ETL/ELT pipelines Background within the financial services industry (investment management experience desirable) Comfortable working with ...

urgently require a Data Platform Engineer with core strengths in Building and Operating Data Platforms, and hands on expertise in Azure Services, Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing and delivering complex cloud solutions. Key Requirements: Proven, hands-on experience of Building and Operating Data … Advanced knowledge of Azure Services Demonstrable experience with Azure Data Factory (ADF), including network configurations and integration runtime options Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts Ability to showcase DevOps and CI/CD capabilities Working knowledge of designing and deploying infrastructure ...

will initially be for 6 months. It is also fully remote. Key Responsibilities • Design, develop, and maintain batch and streaming data pipelines using Databricks (Apache Spark) • Build and optimize ETL/ELT workflows for large-scale structured and unstructured data • Implement Delta Lake architectures (Bronze/Silver/… Gold layers) • Integrate data from multiple sources (databases, APIs, event streams, files) • Optimize Spark jobs for performance, scalability, and cost • Manage data quality, validation, and monitoring • Collaborate with analytics and ML teams to support reporting and model development • Implement CI/CD, version control, and automated testing for data ...

prem data platform aligned to the metals strategy while developing and maintaining scalable data pipelines and analytics infrastructure. Using Hadoop, Big Data, and Spark technologies, you will ensure data quality through automated validation, monitoring, and testing. You will also enable seamless integration across data warehouses and data lakes, contributing … detailed technical design. Hands-on experience configuring multi-node Hadoop clusters, including resource management, security, and performance tuning. Strong Big Data engineering background using Apache Airflow, Spark, dbt, Kafka, and Hadoop ecosystem tools. Knowledge of RDBMS systems (PostgreSQL, SQL Server) and familiarity with NoSQL/distributed databases such ...

data warehousing concepts, and distributed computing. Familiarity with CI/CD, version control, and DevOps practices. Nice-to-Have Experience with streaming technologies (e.g., Spark Structured Streaming, Event Hub, Kafka). Knowledge of MLflow, Unity Catalog, or advanced Databricks features. Exposure to Terraform or other IaC tools. Experience working ...

platform, features, and best practices with Strong understanding of data modeling principles and techniques A solid understanding of big data technologies such as Apache Spark, and knowledge of Hadoop ecosystems. Knowledge of programming languages such as Python, R, or Java is beneficial. Knowledge of ETL/ELT processes ...

about other roles, please contact Nick Mandella at Harnham. KEYWORDS Python, SQL, AWS, GCP, Azure, Cloud, Databricks, Docker, Kubernetes, CI/CD, Terraform, Pyspark, Spark, Kafka, machine learning, statistics, Data Science, Data Scientist, Big Data, Artificial Intelligence, private equity, finance. ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

Solid understanding of data warehousing, lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra). Hands-on experience with cloud platforms (e.g. ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

experience and familiarity with frameworks (e.g. Flask, Django) Experience building and orchestrating ETL pipelines (e.g. Airflow, Luigi, Argo) Exposure to big data technologies (e.g. Spark, Kafka, Hadoop) Experience working in Agile environments Solid understanding of data architecture and data quality best practices ...

data engineering, data platforms and big data technologies Evaluate and improved existing data processes Essential Experiences: Advanced SQL skills Proficient in Databricks, including Spark, Delta Lake and MLflow Experience migrating SSIS to Databricks In depth understanding of ETL/ELT processes Knowledge of data warehousing concepts and architectures Able ...

similarly regulated industry. Strong coding skills in Python or Java (Spring Boot); React experience is a plus. Proficiency with modern data tools: Airflow, Spark, Kafka, dbt, Snowflake or similar. Experience with cloud platforms (AWS, Azure, GCP), containerization (Docker, Kubernetes), and CI/CD. Data Quality: Proven ability to validate ...

data modeling concepts and ETL/ELT processes. Ability to troubleshoot and optimize data workflows. Required Knowledge and Skills: Exposure to Databricks or Spark-based environments. Experience with data governance tools and practices. Experience with JavaScript Full-Stack development environments including MongoDB or similar NoSQL databases Knowledge of version ...

discussions and provide guidance on data engineering best practices. Ability to troubleshoot and optimize data workflows. Required Knowledge and Skills: Experience with Databricks or Spark-based environments, including optimization strategies. Familiarity with data governance tools and frameworks (e.g., Unity Catalog). Experience with JavaScript Full-Stack development environments including ...

plus) Experience with orchestration and workflow tools (Airflow, Cloud Composer, Dagster, etc.) Familiarity with modern data stack components (BigQuery, dbt, Kafka/PubSub, Spark, etc.) Knowledge of data security, access control, and best practices for handling sensitive data Experience collaborating across engineering, analytics, and product teams Strong communication skills ...

sources, analytics, or trading workflows. Exposure to cloud platforms (Azure, AWS, or GCP). Experience hiring, mentoring, or leading small engineering teams. Familiarity with Apache Spark, Kafka, Delta Lake, or similar data technologies. Apply Now to avoid disappointment! To find out more about Huxley, please visit ...

sectors. Strong knowledge of data fabric architectures, metadata management, and data governance principles. Experience with SQL, ETL tools, and big data technologies (e.g., Hadoop, Spark, Kafka). Familiarity with cloud platforms (AWS, Azure, or Google Cloud) and data integration frameworks. Strong problem-solving skills and ability to work ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

TypeScript, JavaScript Machine Learning: TensorFlow, PyTorch, Scikit-learn GenAI/LLMs: OpenAI API, Hugging Face Transformers, LangChain, Langraph, RAG pipelines Data & Analytics: Pandas, NumPy, Spark, Databricks, Hadoop, Vector Databases, Snowflake Model Deployment: MLflow, FastAPI, Flask, Docker, Kubernetes Cloud & Infrastructure: AWS, Azure, GCP CI/CD pipelines (GitLab, Jenkins) Desirable ...

teams Strong background in cloud data platforms and modern data architectures Hands-on expertise with Python , data modelling, ETL, and big data technologies (e.g. Spark) Experience operating in regulated or complex enterprise environments Excellent stakeholder management and communication skills Comfortable working with ambiguity and driving change Nice to have ...

DevOps and broader software engineering standards. Cloud experience (any platform). Previous mentoring experience. Nice to have: Snowflake or Databricks Spark, PySpark, Hadoop or similar big data tooling BI exposure (PowerBI, Tableau, etc.) Interview Process The process is fully structured, transparent, and efficient: Video call - high-level overview ...