1 to 25 of 55 Apache Spark Jobs in London

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

Basic understanding of Python and SQL Familiarity with cloud platforms such as AWS, Microsoft Azure, or Google Cloud Platform (GCP) Knowledge of ETL tools, Apache Spark, Apache Airflow, or Kafka Understanding of relational and NoSQL databases Interest in big data, cloud computing, data warehousing, or machine learning ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

frameworks, tools and applications that make that data available to other teams and systems.Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.What You'll Do: Write highly fault-tolerant Java code within Apache Spark … combined between backend/cloud development and data platform engineering roles5+ years of experience building data platform product(s) or features with (one of) Apache Spark, Flink or Iceberg, or with comparable tools in GCP5+ years of experience programming with Java, Scala or Kotlin.Proven experience utilizing AI technologies ...

standards. Skills Strong experience in SQL and data modeling concepts. Hands-on experience with ETL/ELT development. Proficiency in Python, Scala, or Spark-based data processing. Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools ...

plusAbility to write clean, maintainable, scalable, and robust code in PythonFamiliarity with analytics libraries (e.g., pandas, numpy, matplotlib), distributed computing frameworks (e.g., Spark, Dask), and cloud platforms (e.g., AWS, Azure, GCP)Basic understanding or exposure to containerization technologies such as Docker and Kubernetes would be beneficialExposure to software engineering ...

/ELT processes and data pipeline development Experience working with cloud platforms such as AWS, Azure or GCP Knowledge of technologies such as Spark, Kafka, Airflow or similar Experience with databases, data modelling and data warehouse solutions Understanding of CI/CD, software engineering best practices and secure development ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs. Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

large request volumes, large data volumes, or both.Deep expertise in distributed data systems: streaming (Kafka or similar), batch and ELT/ETL frameworks (Spark, Flink, dbt, Airflow or Argo Workflows), and modern lakehouse or warehouse technologies.Strong production Python experience and awareness of other relevant languages (Java/Kotlin) sufficient ...

Market Risk, XVA, CCR, FRTB, portfolio risk, fixed income analytics, securitised products, liquidity risk, or managed risk-as-a-service solutions.Familiarity with Snowflake, Databricks, Spark, AWS, Azure, Confluence, Azure DevOps, Git, Tableau, Power BI, message queues, or comparable tools and platforms.Knowledge of market data vendors, data mastering, golden-source ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. Experience ...

progression taking customers from EDW to the full capabilities of the Data Intelligence Platform.Demonstrated competence in the Lakehouse architecture including hands-on experience with Apache Spark, Python and SQLExcellent communication skills; both written and verbalPublic Cloud Architecting: Hands-on experience designing and implementing big data architectures within major ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

stakeholders, and business leaders.Strong presentation, demo, and storytelling skills, with the ability to explain complex technical concepts clearly and credibly.Familiarity with SQL, Python, R, Apache Spark, or similar technologies is a plus.Experience with Alteryx or similar analytics platforms is highly desirable.Ability to work independently in a fast-paced ...

security controls (RBAC, encryption, auditing, lineage). Essential Requirements Significant Databricks experience required, including production-scale deployments and optimisation. Strong hands-on capability with Spark/SQL, Delta Lake, orchestration, and data engineering patterns. Prior experience within a consulting practice with exposure to both pre-sales and delivery leading ...

Senior Databricks Architect - London (Hybrid) - Databricks - Spark - PySpark - AWS - Up to £115,000 + Bonus Our client, a large-scale organisation undergoing significant investment in data and cloud technologies, is seeking an experienced Senior Databricks Architect to join its expanding Data & AI practice. This is an exciting opportunity … real-time data pipelines from legacy and modern source systems Implementing real-time and event-driven data processing solutions Developing data transformation frameworks using Spark, PySpark and SQL Designing Lakehouse architectures using Databricks and Medallion Architecture principles Creating scalable, reusable data products to support modern data operating models Integrating ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

+ ETL/ELT experience Exposure to DBT Knowledge of Python/T-SQL Exposure to cloud platforms (Azure preferred) Nice to have: Databricks, Spark, CI/CD, big data This is a great opportunity to join a high-impact data team in a global organisation and shape ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

services, capital markets or FinTech. Additional AWS services such as SQS, SNS, Step Functions, EventBridge or Secrets Manager. Big data technologies including DuckDB, pandas, Spark, Databricks or Snowflake. AI/ML technologies including vector databases, semantic search or LLM orchestration frameworks. Continuous delivery and modern development workflows. Data science ...