51 to 75 of 162 Apache Spark Jobs in London

Senior Databricks Architect - London (Hybrid) - Databricks - Spark - PySpark - AWS - Up to £115,000 + Bonus Our client, a large-scale organisation undergoing significant investment in data and cloud technologies, is seeking an experienced Senior Databricks Architect to join its expanding Data & AI practice. This is an exciting opportunity … real-time data pipelines from legacy and modern source systems Implementing real-time and event-driven data processing solutions Developing data transformation frameworks using Spark, PySpark and SQL Designing Lakehouse architectures using Databricks and Medallion Architecture principles Creating scalable, reusable data products to support modern data operating models Integrating ...

ability to help refine problem statements and propose solutions taking effort-impact-scalability tradeoff into account.Some skills that will make you stand out: Apache Spark, Iceberg, Kafka, dbtScikit-Learn, XGBoost, PyTorch, MLFlow,, GraphFrames, RayAWS (S3, EMR, SageMaker, Lakeformation), Terraform, Docker, GitHub CI/CDKnowledge Graphs (+ RAG), graph ...

Github integration and automation). Experience with scripting languages such as Python or R. Working knowledge of message queuing and stream processing. Experience with Apache Spark or Similar Technologies. Experience with Agile and Scrum Technologies. Familiarity with dbt and Airflow is an advantage. Experience working in a start ...

complex pipelines and data models in a cloud environment Strong proficiency in SQL; solid Python and/or PySpark capabilityDemonstrable experience with Databricks, Apache Spark, and Delta Lake architectureProven ability to design and own enterprise-grade canonical data models across multiple upstream source systemsExperience integrating data from APIs ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

systems Comfortable building from scratch in a fast-moving, product-led environment Experience with vector databases and vector search in modern AI systems Desirable: Spark, Airflow, Kafka, Elasticsearch/OpenSearch, event-driven architectures Cloud experience (AWS, GCP or Azure), GCP is a plus Why join High impact and ownership ...

machine learning, model development, and data engineering, with an appreciation of responsible AI principles.Experience with cloud environments (Azure, AWS, GCP) and tools such as Spark, Hive, and Redshift.Demonstrated ability to lead cross-functional teams and mentor junior practitioners.Ability to communicate complex technical concepts clearly to non-technical audiences ...

monitoring, controls and auditability) and how to embed them pragmatically within client delivery.Familiarity with large-scale data processing and storage technologies (e.g., SQL, Spark).Excellent communication and stakeholder management skills, including the ability to translate complex technical concepts for diverse business and technical audiences.Connect to your business - Technology ...

+ ETL/ELT experience Knowledge of Python/T-SQL Exposure to DBT Exposure to cloud platforms (Azure preferred) Nice to have: Databricks, Spark, CI/CD, big data This is a great opportunity to join a high-impact data team in a global organisation and shape ...

data lake/warehouse/hub built in GCP. You are confident using the full suite of Google data products, IAC, CI/CD, Spark and Kafka. Our core toolbox includes Google Cloud Big Data technologies, Scala, Java & Python, Jenkins amongst others. We value first principles reasoning to select ...

large-scale datasetsPreferred QualificationsKnowledge or interest in investment banking or financial instrumentsExperience with big data concepts, such as Lakehouse, ETL data pipelines (e.g. Spark, AWS Glue)Experience with near real time and event-based systems like Kafka ABOUT GOLDMAN SACHS At Goldman Sachs, we commit our people, capital ...

deploying machine learning models into production environments (MLOps experience is advantageous).Hands‐on experience with big‐data or distributed computing tools such as Spark or Databricks.Familiarity with visualisation tools such as Power BI, Tableau, or Plotly.Industry experience in sectors such as retail, finance, healthcare, or similar (customisable).Key CompetenciesStrong ...

pacing or auction systemsExcellent communication skills, able to explain complex technical systems to commercial, operations and senior leadership teams.Experience with Google Cloud, Docker, Kafka, Spark, Airflow, ElasticSearch, ClickHouse or similar production infrastructureExperience owning product roadmaps, release governance or cross-functional delivery across distributed teams and time zones.What ...

Generative AI and Agentic AI, applied within enterprise environments.Experience with energy system modelling, grid analytics, demand forecasting or energy market data.Familiarity with distributed computing, Spark, PySpark or cloud-native data platforms.Experience developing models for planning, scenario analysis, operational decision support, model governance or regulatory reporting.Why Join CapcoDeliver high-impact ...

including feature engineering, model training, and evaluation metrics Familiarity with ML frameworks, e.g., PyTorch, TensorFlow, PySpark Experience with large scale data processing systems, e.g., Spark Work in a way that works for youWe promote a healthy work/life balance across the organization. We offer an appealing working prospect ...

practices such as MLflow, Airflow, model monitoring, API monitoring, data validation, data drift detection, autoscaling and access management.Strong Python experience, with helpful knowledge of Spark or PySpark for working with large-scale data and machine learning systems.An understanding of feature stores and related data technologies used to support operational ...

e.g., AWS, Azure, GCP or equivalent) and their AI/ML services.Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop) is a plus.Experience with MLOps tools and best practices for model deployment, monitoring, and governance.Education and Experience:Bachelor's or Master ...

libraries (e.g., scikit-learn, pandas, PyTorch or TensorFlow). Practical experience with Databricks, including MLflow for experiment tracking and model management, and Spark-based data processing. Familiarity with Generative AI concepts and tooling, including working with LLMs, prompt engineering, RAG patterns, and vector search. A solid understanding of MLOps ...

MLOps/LLMOps principles, including CI/CD for ML, model monitoring, and governance frameworks.Proficiency with data processing and storage technologies (eg SQL, Spark)Strong stakeholder management and communication skills, with proven ability to translate complex AI concepts for diverse audiences.SUSTAINABLE BUSINESSPassion and curiosity to tackle some of industry ...

MLOps/LLMOps principles, including CI/CD for ML, model monitoring, and governance frameworks.Proficiency with large-scale data processing and storage technologies (SQL, Spark, Hadoop) is a plus.Excellent stakeholder management and communication skills, with proven ability to translate complex AI concepts for diverse audiences.You can understand client challenges ...

using Python, PySpark, and dbtBuild and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observabilityDevelop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformationsModel and transform data for analytics using SQL and dbt to support business intelligence … scale distributed data processing and performance tuningHands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar)Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain).Hands-on experience working with Distributed Spark based systemsExperience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external clients across … variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCommunication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences.Core Traits: Passion for lifelong learning ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain).Hands-on experience working with Distributed Spark based systemsExperience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external clients across … variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCommunication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences.Core Traits: Passion for lifelong learning ...

language processing, including vector databases, fine-tuning LLMs, and deploying LLMs with tools such as HuggingFace, Langchain, and OpenAIHands-on experience working with Distributed Spark based systems.Graduate degree in a quantitative discipline (Computer Science, Engineering, Statistics, Operations Research, etc.) or equivalent practical experienceExperience communicating and teaching technical concepts … driving our values through ML[Preferred] 2+ years customer-facing experience in a pre-sales or post-sales role[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCan meet expectations for technical training and role-specific outcomes within 3 months of hireCan travel ...

showing” rather than “telling”, preferring to inspire stakeholders through built examples.Technical Expertise: Deep experience in core data & AI technologies, including languages (Python, SQL), frameworks (Apache Spark) and platforms (Databricks, etc.)Collaboration: You can orchestrate diverse teams to achieve a common goal and explain complex tech to non-technical ...