1 to 25 of 38 Apache Spark Jobs in London

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

standards. Skills Strong experience in SQL and data modeling concepts. Hands-on experience with ETL/ELT development. Proficiency in Python, Scala, or Spark-based data processing. Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Reporting ...

/ELT processes and data pipeline development Experience working with cloud platforms such as AWS, Azure or GCP Knowledge of technologies such as Spark, Kafka, Airflow or similar Experience with databases, data modelling and data warehouse solutions Understanding of CI/CD, software engineering best practices and secure development ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs. Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

optimising ETL/ELT pipelines. Strong understanding of Lakehouse, Data Warehouse, and Data Lake architectures. Experience with Azure Data Factory, Azure Synapse, and Apache Spark. Knowledge of data modelling, semantic models, and enterprise reporting solutions. Experience implementing data governance, security, and quality frameworks. Strong understanding of modern data engineering ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. Experience ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

security controls (RBAC, encryption, auditing, lineage). Essential Requirements Significant Databricks experience required, including production-scale deployments and optimisation. Strong hands-on capability with Spark/SQL, Delta Lake, orchestration, and data engineering patterns. Prior experience within a consulting practice with exposure to both pre-sales and delivery leading ...

Senior Databricks Architect - London (Hybrid) - Databricks - Spark - PySpark - AWS - Up to £115,000 + Bonus Our client, a large-scale organisation undergoing significant investment in data and cloud technologies, is seeking an experienced Senior Databricks Architect to join its expanding Data & AI practice. This is an exciting opportunity … real-time data pipelines from legacy and modern source systems Implementing real-time and event-driven data processing solutions Developing data transformation frameworks using Spark, PySpark and SQL Designing Lakehouse architectures using Databricks and Medallion Architecture principles Creating scalable, reusable data products to support modern data operating models Integrating ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

services, capital markets or FinTech. Additional AWS services such as SQS, SNS, Step Functions, EventBridge or Secrets Manager. Big data technologies including DuckDB, pandas, Spark, Databricks or Snowflake. AI/ML technologies including vector databases, semantic search or LLM orchestration frameworks. Continuous delivery and modern development workflows. Data science ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

machine learning and data science frameworks. Hands-on experience with large language models, generative AI, prompt engineering, and RAG architectures. Experience with Databricks, MLflow, Spark, and cloud-based AI platforms (Azure preferred). Strong understanding of MLOps, software engineering best practices, and production deployment pipelines. Experience leading technical teams ...

Weaviate or pgvector Docker, Kubernetes, ECS or containerised deployments Infrastructure as Code tools including Terraform or CloudFormation Large-scale data processing technologies such as Spark or Dask Experience working within regulated or highly secure environments Knowledge of data governance and compliance frameworks Apply for This Machine Learning Engineer Opportunity ...

Hands-on experience with ETL/ELT pipelines, data engineering, and large-scale data processing Experience with containerisation (Docker) and scalable data systems (e.g. Spark, Kafka) Nice-to-Have Experience with AWS SageMaker or similar managed ML platforms Background in safety-critical or regulated industries (aerospace, aviation, or similar ...

large-scale analytics platforms Expert knowledge of Quantexa, Entity Resolution (ER), Scoring Framework, Contextual Network Generation Expertise in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Strong stakeholder management and communication skills - capable of interfacing with both business and technical audiences. The work is heavily weighted toward ...

Python and MLflow Strong AWS/cloud-native experience ( ideally with SageMaker exposure) React frontend proficiency Docker, Kafka, and large-scale data systems (e.g. Spark) experience Strong leadership presence and senior stakeholder communication skills If you have the technical expertise, leadership experience and ambition to make an impact ...

Python and MLflow Strong AWS/cloud-native experience ( ideally with SageMaker exposure) React frontend proficiency Docker, Kafka, and large-scale data systems (e.g. Spark) experience Strong leadership presence and senior stakeholder communication skills If you have the technical expertise, leadership experience and ambition to make an impact ...