16 of 16 Apache Spark Jobs in the City of London

flexibility to attend key meetings onsite. Required: Good knowledge and experience using programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful: R, Apache Spark Very Useful: Python, SQL, JSON, LakeFS ...

data sets can be effectively combined and analysed. They can also utilise programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful - R, Apache Spark Very useful - Python, SQL, JSON, LakeFS. ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across Cloud (Azure, AWS, GCP) and Big Data (Spark, Kafka) stacks. Collaboration: Work closely with business teams to define data flows and ensure security protocols (encryption/authentication) are met. Technical Toolkit Languages ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

processes Our ideal candidate has: Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms ...

years of Scala experience Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs ...

influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing … Engineer or Solutions Architect certification (desirable). Databricks Certified Data Engineer or Machine Learning Associate (desirable). Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable). Knowledge of machine learning and AI on Databricks (desirable). Benefits of working at Canada Life We believe in recognising ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

data science or statistical role. Expertise in advanced analytics techniques including machine learning, optimisation and segmentation. Strong proficiency in Python, SQL and Spark SQL. Experience working with large datasets and distributed computing tools such as Azure Synapse and Snowflake. Familiarity with data transformation tools (Databricks, Azure Data Factory ...

architectures. Familiarity with document processing pipelines, chunking strategies, and semantic search patterns. Familiarity with data pipeline and orchestration tools (e.g., Airflow, dbt, Spark, Kafka, cloud-native ETL tools) and batch vs. streaming patterns. Understanding of data governance (catalog, lineage, security, RBAC, masking, compliance requirements like GDPR/CCPA). … data science and ML workflows (feature engineering, experimentation, model training/deployment, RAG pipeline development, prompt engineering) and tools/languages such as Python, Spark, notebooks, and ML frameworks (e.g., scikit-learn, MLflow, TensorFlow/PyTorch, LangChain, LlamaIndex at a conceptual level). Consulting Skills Skilled at asking ...

RNNs, Transformers) and when to apply them Strong SQL and data manipulation capabilities Experience working with large-scale datasets and distributed compute frameworks (e.g., Spark) Demonstrated ability to independently lead technical ML solution design Experience working in client-facing delivery environments Exposure to cloud-based ML platforms (AWS, Azure ...

deploy scalable, secure data solutions using Azure Databricks, Data Factory and Data Lake Storage. Develop and optimise advanced data pipelines with Python, SQL, Spark/PySpark and Delta Lake. Champion strong data quality, governance and observability practices. Modernise legacy systems and support large-scale migrations into Azure. Provide technical … Minimum Criteria Extensive experience with Azure services including Azure Databricks, Azure Data Lake Storage, and Azure Data Factory. Advanced proficiency in SQL, Python, and Spark (PySpark), with a strong focus on performance optimization and distributed processing. Proven experience in CI/CD practices using industry-standard tools (e.g., GitHub ...

engineering practices (clean code, testing, version control, code review) Proficient with large-scale data : strong SQL skills and familiarity with distributed data processing (e.g., Spark, Hive) for building training datasets, features, and analytical views Leadership & Collaboration Proven ability to lead through influence : aligning cross-functional stakeholders on problem definitions ...