101 to 125 of 260 Apache Spark Jobs in the UK

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

databases, RAG, grounding, reranking and evaluation Experience with NLP techniques such as text classification, entity extraction, semantic similarity,summarisationand information retrieval Experience using Databricks, Spark andlakehouseconcepts for large-scale data preparation,experimentationand model development Familiarity with Microsoft Fabric, Azure AI services, Azure Machine Learning or comparable cloud-based ...

databases, RAG, grounding, reranking and evaluation Experience with NLP techniques such as text classification, entity extraction, semantic similarity,summarisationand information retrieval Experience using Databricks, Spark andlakehouseconcepts for large-scale data preparation,experimentationand model development Familiarity with Microsoft Fabric, Azure AI services, Azure Machine Learning or comparable cloud-based ...

databases, RAG, grounding, reranking and evaluation Experience with NLP techniques such as text classification, entity extraction, semantic similarity,summarisationand information retrieval Experience using Databricks, Spark andlakehouseconcepts for large-scale data preparation,experimentationand model development Familiarity with Microsoft Fabric, Azure AI services, Azure Machine Learning or comparable cloud-based ...

services, capital markets or FinTech. Additional AWS services such as SQS, SNS, Step Functions, EventBridge or Secrets Manager. Big data technologies including DuckDB, pandas, Spark, Databricks or Snowflake. AI/ML technologies including vector databases, semantic search or LLM orchestration frameworks. Continuous delivery and modern development workflows. Data science ...

stakeholders. Strong system‐level problem‐solving skills with the ability to make architectural trade‐offs. Experience with modern analytics tools and platforms such as Spark, data lakes, and similar. Hands‐on experience with LLMs, AI prompt design, and building production‐grade AI applications. Advantage Previous experience as a DBA. ...

experience applying them with packages like Pandas, sci-kit-learn, SciPy) is highly beneficial Experience processing big data, ideally in a Hadoop/Spark environment, would be beneficial Proven experience implementing CI/CD and DevOps practices, with the ability to improve deployment reliability, testing standards, and release processes ...

experience applying them with packages like Pandas, sci-kit-learn, SciPy) is highly beneficial Experience processing big data, ideally in a Hadoop/Spark environment, would be beneficial Proven experience implementing CI/CD and DevOps practices, with the ability to improve deployment reliability, testing standards, and release processes ...

value databases Strong knowledge of TDD, Unit, and automated test paradigms Experience with SQL and RDBMS solutions Experience with large-scale data processing (Spark) Headquartered in San Francisco, CA, Backstory is backed by Y Combinator and Silicon Valley’s top investors, including ICONIQ Capital, Andreessen Horowitz, Lightspeed Venture Partners ...

machine learning development (training and/or deploying models at scale). Outstanding programming skills in Python. Experience with distributed data processing frameworks (e.g., Spark). Experience with deep learning/NLP frameworks (e.g., PyTorch, Hugging Face). Solid understanding of software engineering principles and ML lifecycle practices (MLOps ...

encouraged to talk about it Experience in financial services,insuranceor another regulated sector Experience deploying real‐time or streaming ML models (e.g.Kafka, Flink, Spark Structured Streaming) Passion for automation, tooling, and building reusable systems Interest in responsible AI and ML model governance Why Compare the Market ...

both OnPrem and Cloud* Experience with processing large datasets as well as streaming data and unstructured data sources including Cassandra HBase Kafka and Spark* Strong ETL and data pipeline experience particularly with modern big data toolsets such as Airflow and Nifi or equivalent* Strong programming background in either Java ...

design scalable Lakehouse solutions supporting critical business data. What you'll be doing Leading Azure Databricks architecture Designing enterprise Lakehouse solutions Building scalable Spark and Delta Lake pipelines Designing streaming and batch data platforms Driving engineering best practice Mentoring engineers Working closely with senior business stakeholders Defining future-state … data architecture Technical Environment Azure Databricks Apache Spark/PySpark Delta Lake Azure Data Factory Azure Data Lake Gen2 Azure Event Hubs Azure Synapse Python SQL Terraform CI/CD Medallion Architecture We're looking for Expert Databricks experience Strong Azure Data Platform experience Enterprise Data Architecture Hands ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine‐tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands‐on experience working with Distributed Spark based systems Experience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer‐facing experience would be preferred Preferred: Experience working with Apache Spark (TM) to process large‐scale distributed datasets Communication: Proven ability to communicate and teach complex technical concepts to both technical and non‐technical audiences. ...

support data pipelines, ensuring they are production-ready and meet data quality standards. You will work with technologies such as Python, SQL, and Apache Spark, helping drive analytics and operational decision-making. Ideal candidates possess strong programming skills, a relevant degree, and a collaborative spirit, as you will ...

Codebase ownership Technical Application ownership Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins, Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

rather than “telling”, preferring to inspire stakeholders through built examples. Technical Expertise: Deep experience in core data & AI technologies, including languages (Python, SQL), frameworks (Apache Spark) and platforms (Databricks, etc.) Collaboration: You can orchestrate diverse teams to achieve a common goal and explain complex tech to non-technical ...

Experience with Infrastructure-as-Code tools such as Terraform Knowledge of DevOps, CI/CD, platform automation, and cloud‐native operational practices Familiarity with Apache Spark, dbt, and modern data integration technologies Delivery Experience Experience leading architecture for enterprise-wide Databricks implementations and cloud data modernization initiatives Proven ...

ideal candidate will have a strong programming background in Python or Java, knowledge of SQL, and experience with data processing frameworks such as Apache Spark. Responsibilities include enhancing data pipelines, ensuring data quality, and collaborating with engineering teams. The position is located in London and offers significant growth opportunities ...

Strong commercial experience in data engineering and modern data platforms. Advanced programming skills in Java or Python. Expertise working with technologies such as Kafka, Spark, Flink or similar distributed data frameworks. Experience delivering solutions within AWS or other major cloud platforms. Strong understanding of data architecture, scalability and performance ...