1 to 25 of 83 Apache Jobs in London

detect threats, analyze patterns, and respond to incidents in real-time. You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teams Build and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructure Design event-driven systems leveraging Kafka for high-throughput, low-latency message streaming ...

3+ years of experience with Spark, Databricks, Snowflake, Hadoop, or other big data technologies . Nice to Have: Experience with orchestration tools such as Apache Airflow, Azure Data Factory, or AWS Glue. Knowledge of streaming technologies such as Kafka, Kinesis, or Pub/Sub. Experience with data warehouse solutions ...

understanding of Quantexa components such as: Entity Resolution (ER) Scoring Framework Contextual Network Generation UI Configuration Strong background in Big Data technologies, including:Scala, Apache Spark, Hadoop, ElasticSearch Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

code. Modern Data & Cloud Practices : Experience with modern data engineering tooling and cloud platforms (ideally AWS and Databricks), and familiarity with distributed processing (e.g. Apache Spark). Exposure to infrastructure as code (such as Terraform and Databricks Asset Bundles) is a plus. Engineering Mindset : Understanding of testing, version control ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

Experience with Infrastructure-as-Code tools such as Terraform Knowledge of DevOps, CI/CD, platform automation, and cloud‐native operational practices Familiarity with Apache Spark, dbt, and modern data integration technologies Delivery Experience Experience leading architecture for enterprise-wide Databricks implementations and cloud data modernization initiatives Proven success ...

engineers and support code reviews, design reviews, and engineering best practices. Your Skills: Strong hands-on experience with Databricks on AWS. Strong experience with Apache Spark/PySpark. Excellent programming skills in Python. Strong SQL skills including complex queries, joins, CTEs, window functions, and query optimization. Experience with Spark ...

market data engineering with proven experience building reliable, scalable data platforms. Our non-negotiable requirements are: Expert Python Strong SQL Scalable data pipeline development Apache Parquet Market data normalisation Automated data quality controls Deep tick and market data expertise Experience in MLOps is advantageous but not essential. We welcome ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

ability to help refine problem statements and propose solutions taking effort-impact-scalability tradeoff into account. Some skills that will make you stand out: Apache Spark, Iceberg, Kafka, dbt Scikit-Learn, XGBoost, PyTorch, MLFlow,, GraphFrames, Ray AWS (S3, EMR, SageMaker, Lakeformation), Terraform, Docker, GitHub CI/CD Knowledge Graphs ...

Ideally will be a bonus if you have the following... Experience or knowledge of FluxCD, GitOps workflow, Appgate, MongoDB, Airflow (MWAA), Snowflake Experience in Apache Airflow (MWAA), AWS Backup and S3 versioning for business continuity. Programming experience in C#, Python and/or SQL Experience in Keycloak or other ...

Infra security, compliance and governance • Familiarity with CNCF ecosystem and building platforms on top of Kubernetes • Container orchestration tools, e.g. Docker Swarm/Apache Mesos • Knowledge of Service Mesh technologies, e.g. Istio/Linkerd • Experience of monitoring tools, e.g. Prometheus, Grafana, ELK • Rounded problem-solving skills - troubleshoot and resolve ...

with data scientists, analysts, and business stakeholders. Strong understanding of data platform reliability, performance, scalability, and cost optimisation. Experience with technologies such as ClickHouse, Apache Iceberg, BigQuery, dbt, Elasticsearch, Druid, GCP, or AWS is highly desirable. If you're an experienced Senior Data Engineer looking to lead complex data ...

innovation-led working environment Key Experience Proven experience as a Data Architect, Databricks Architect, Lead Data Engineer or Solutions Architect Strong expertise in Databricks, Apache Spark, PySpark and SQL Experience designing and implementing modern data lake or lakehouse architectures Hands-on experience with Kafka Strong understanding of AWS data ...

skills using Python. (Nice to have) \n *Proficient with Pandas for handling and transforming complex datasets. (nice to have) \n *Hands-on experience with Apache Airflow for data orchestration and pipeline scheduling. \n Experience of mentoring junior developers. \n Preferred Qualifications \n *Experience in cloud-based environments (especially ...

reconciling source and target data. Experience executing functional and data validation testing. Agile experience. Technical Environment Microsoft Fabric Azure Data Factory SQL Python PySpark Apache Spark Desirable Skills Experience introducing test automation. Data quality frameworks. Reporting and analytics testing. Azure DevOps. Contract Details UK-based London office 4 days ...

statistics skills: distributions, statistical testing, regression analysis Cloud management and deployment – A strong background in different cloud platforms like Amazon AWS, Microsoft Azure, OpenStack Apache, and Google Cloud Platform Databases – A familiarity of working with different databases such as Postgres, MySQL, NoSQL (e.g., MongoDB), Redis is recommended. Web technologies ...

Understanding of Azure data services and API-based integration. Next.js 14, FastAPI Python, Pipeline orchestration, Data transformation, Data quality, Casbin RBAC, Kuzu knowledge graph, Apache Iceberg, MCP AI server integration and ODCS v2 data contracts layer. Experience in establishing the infrastructure, pipelines, processes, and governance required for production environment ...

production, providing subject matter expertise on the .Net stack and contributing to technical design discussions. You'll use a range of technology including Apache Flink with Java for large scale data processing and will be able to assess and recommend new and emerging technologies, using the best tool ...

engineering mindset and experience building production-grade systems Experience developing ETL processes and large-scale financial data pipelines Strong Azure experience Docker and Kubernetes Apache Airflow GitHub and CI/CD (GitHub Actions) Experience building cloud-native research or production platforms Experience working with market data, security master data ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. EEO Statement At Databricks, we are committed to fostering a diverse and inclusive culture where everyone can excel. ...

data, analytics and AI. Databricks is headquartered in San Francisco, with offices around the globe and was founded by the original creators of Lakehouse, Apache SparkTM, Delta Lake and MLflow. Benefits At Databricks, we strive to provide comprehensive benefits and perks that meet the needs ...

Collaborate with cross‐functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial domains ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What You Will … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise ETL/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high … commercial experience developing data engineering solutions using Python . Extensive hands-on experience with Databricks , including Workflows, Notebooks and Delta Lake. Strong experience with Apache Spark/PySpark . Proven experience building scalable ETL/ELT pipelines. Strong SQL and data modelling skills. Experience designing modern Lakehouse architectures. Experience ...