251 to 275 of 305 Apache Spark Jobs in London

DescriptionRole –Senior Databricks ArchitectTechnology – Databricks (Spark, PySpark/SQL) and data lake architecturesLocation – London, UKCompensation – Competitive (including bonus)Job DescriptionWe are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role focuses on integrating legacy … systems with cloud-native platforms to support analytics, reporting, and domain-owned data products.Primary Skills• Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures.• Hands-on experience with streaming technologies, especially Kafka/Confluent Kafka.• Experience integrating legacy data sources such as DB2 (mainframe ...

build and enhance our Databricks platform, ensuring it remains secure, scalable and reliable. Develop and maintain batch and real-time data pipelines using Databricks, Spark and Delta Lake. Contribute to platform architecture, engineering standards and best practices across the data estate. Collaborate with Product, Analytics and Data Science teams … improvement across the engineering team. Our ideal candidate Proven experience building and supporting scalable, production-grade data platforms. Strong hands-on expertise with Databricks, Spark, Delta Lake, Python and SQL. Good understanding of cloud-based data engineering, ideally within Azure environments. Experience working with Infrastructure as Code tools such ...

degree, or experience in a professional field or military- Experience as technical specialist in design and architecture- Experience in database (eg. SQL, NoSQL, Hadoop, Spark, Kafka, Kinesis)- Experience in consulting, design and implementation of serverless distributed solutions- Experience in cloud based solution (AWS or equivalent), system, network and operating … military settings- Bachelor's degree, or experience as technical specialist in design and architecture- Experience in Database like NoSQL, or experience in Hive/Spark/Hbase/Yarn and experience in any Bigdata architecture- Bachelor's degree in business administration, finance, economics, computer science, data science, engineering ...

enhance solution quality while applying sound engineering judgement and responsible AI practices.What You'll DoDesign and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services.Build real-time, event-driven solutions using technologies such as Kafka and Azure Event Hubs, applying AI-enabled … responsibly to improve productivity while maintaining appropriate human oversight, governance, and solution quality.Bonus Points ForDevelopment experience using Scala or Java.Familiarity with Cloudera, Hadoop, Hive, Spark, and related distributed data ecosystems.Experience working with sensitive data and applying data privacy regulations, including GDPR.Curiosity to explore new technologies, including responsible ...

translate them into solutions.Key responsibilities:Building and maintaining data pipelines and services under the guidance of senior engineers.Working with technologies such as Python, SQL, Spark, and Airflow to process large-scale data.Using AI-assisted development tools such as Claude Code, Copilot, or Codex as part of your workflow.Contributing … Python, SQL, and Spark.Basic understanding of data engineering concepts including batch processing and relational databases.Some hands-on experience with data pipeline tools; exposure to Spark or Airflow is a plus.Familiarity with cloud platforms (preferably AWS); production experience is not required.A collaborative mindset and willingness to learn and communicate openly ...

enhance solution quality while applying sound engineering judgement and responsible AI practices. What You'll DoDesign and develop robust data pipelines using Delta Lake, Spark Structured Streaming, Unity Catalog, and Azure-native services. Build real-time, event-driven solutions using technologies such as Kafka and Azure Event Hubs, applying … improve productivity while maintaining appropriate human oversight, governance, and solution quality. Bonus Points ForDevelopment experience using Scala or Java. Familiarity with Cloudera, Hadoop, Hive, Spark, and related distributed data ecosystems. Experience working with sensitive data and applying data privacy regulations, including GDPR.Curiosity to explore new technologies, including responsible ...

appropriate technologies for scalable analytics, trading strategy development, tick or time-series data, workflow automation and AI/ML-enabled use cases.• Prefer Databricks, Spark and modern lakehouse patterns for scalable analytics where suitable, while recognising transferable experience with Snowflake and other enterprise data platforms.• Ensure delivery choices balance … define OKRs, shape roadmaps and deliver enterprise data or analytics products.• Hands-on understanding of data technologies and modern analytics platforms, preferably Databricks/Spark and lakehouse patterns; Snowflake experience is relevant and transferable but Databricks experience is preferred.• Experience with Python, SQL and production analytics workflows; familiarity with ...

DescriptionAs a Junior Software Engineer (Data Products) at Yapily, you will help build and scale ourmodern, next-generation data platform. This is a junior level role perfect for someone with astrong computer science foundation who ...

focusing on backend systems for large‐scale data storage, streaming, and warehousing. Experience with database technologies (MongoDB, Postgres), streaming/processing solutions (Kinesis, Flink, Spark), indexing/caching (ElasticSearch, Redis), and query engines (Trino, Presto, Snowflake). Track record of mentoring and leading teams in successful projects. Excellent communication ...

engineering multi-site fail-over solutionsCandidates Must Have:4 years commercial Linux CentOS or Debian administration experiencePuppet configuration management or equivalent automation experienceHadoop/Spark Experience of security standards, best practices and principles.Strong scripting in at least one language ideally Ruby, PythonMaven, JenkinsMonitoring and alerting in Sensu/Graphite ...

related to building large-scale data storage, streaming, and warehousing systems. Experience in various database technologies (MongoDB, Postgres), streaming/processing solutions (Kinesis, Flink, Spark), indexing/caching (ElasticSearch, Redis), and various data query engines (Trino, Presto, Snowflake, etc.). Show a track record of mentoring and leading teams ...

specifically related to building large-scale data storage, streaming, and warehousing systems.Experience in various database technologies (MongoDB, Postgres), streaming/processing solutions (Kinesis, Flink, Spark), indexing/caching (ElasticSearch, Redis), and various data query engines (Trino, Presto, Snowflake, etc.).Show a track record of mentoring and leading teams ...

research ideas to productionTrack record of promoting statistical rigour and experimental best practice across a teamPrior experience with large datasets and distributed computing (e.g. spark, hadoop or other map-reduce tech)Experience working in a fast-moving tech company or startupThis is a full-time role based ...

society's experience with software. Come join one of the fastest‐growing startups, supported by best‐in‐class institutions like Battery Ventures, Salesforce Ventures, Spark Capital and Meritech. Pendo Core Values: Bias to Act, Hone Your Craft, The Team is Pendo, and Maniacal Focus. Location: Pendo is a hybrid ...

society's experience with software. Come join one of the fastest-growing startups, supported by best-in-class institutions like Battery Ventures, Salesforce Ventures, Spark Capital and Meritech.Pendo Core Values: Bias to Act, Hone Your Craft, The Team is Pendo, and Maniacal Focus.Location: Pendo is a hybrid culture. ...

gambling industry. Your work will involve shaping lakehouse and data-lake paradigms, leveraging Parquet and Iceberg for high-performance data storage, and adopting scalable Spark frameworks for large-scale processing. Ultimately, you will enable self-service analytics, personalised user experiences, and an AI-first approach to data-driven decision … DevOps practices, ensuring compliance with FDJ United’s security and regulatory standards. Modernising Data Pipelines & Workflows: Implement Scalable Frameworks, orchestrate large-scale pipelines using Spark, Docker, Kubernetes, and other DevOps tools. Accelerate Delivery: Apply standardised processes and automated testing to streamline development and reduce time-to-market while managing ...

data architectures and engineering standards that support analytics, AI and product innovation. Lead the delivery of batch and real-time data pipelines using Databricks, Spark and Delta Lake. Coach and mentor a team of engineers, promoting technical excellence and continuous improvement. Work closely with Product, Analytics and Data Science … through data. Our ideal candidate Proven experience leading the design and delivery of scalable, production-grade data platforms. Deep hands-on expertise with Databricks, Spark, Delta Lake, Python and SQL. Strong understanding of cloud-native data engineering, including Azure and Infrastructure as Code (Terraform, Bicep or similar). Expertise ...

Help shape the technical roadmap for data engineering across Elliptic. Leverage and deploy AI and agentic systems to operate more effectively. Tech environment Scala | Spark | Databricks | AWS | Airflow | Kubernetes | Terraform | Functional Programming You will thrive here if you Enjoy end‐to‐end ownership, from writing the code to architecture … programming. Ability to design, build, and maintain distributed data systems in a cloud‐based environment. Hands‐on experience with big data frameworks such as Spark or Databricks. Knowledge of cloud infrastructure (AWS, GCP, or Azure). Judgement to balance scalability, performance, and maintainability. Experience with data modelling and workflow ...

platform capabilities for data ingestion, transformation, orchestration, and processing across batch and real-time workloads. Own the integration strategy across tools such as dbt, Spark and Beam Lead the implementation and adoption of data quality, observability, security, testing and deployment including creating and sharing standards Partner closely with Analytics … supporting others Hands on experience transitioning to an AI-native Data Engineering landscape A strong Python and SQL engineer with experience using dbt, Spark and Terraform Strong experience working with GCP (or a comparable cloud platform), Infrastructure as Code, and CI/CD practices. Deep understanding of data storage ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache SparkTM and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/AI models ...

recommendations)Strong focus on data quality and correctness, and how upstream data impacts downstream models and user experienceStrong SQL skills for massive datasets (BigQuery, Spark)Cloud platform experience (AWS/GCP) and containerization (Docker, Kubernetes)Experience with distributed data processing and ETL pipelines (Airflow, Spark)Familiarity with ...

bringing strong opinions, updating them when better ideas emerge, and communicating decisions clearly to technical and non-technical stakeholders alikeQualificationsDeep proficiency in Python and Spark for building and optimizing large-scale data pipelines, with a strong grasp of data structuring for performance and efficiencyHands-on experience with Airflow … orchestrating complex data workflows, and Apache Iceberg for table format management at scaleExtensive AWS experience including EKS, Docker, and Kubernetes; comfortable owning infrastructure deployment and scaling decisionsCI/CD fluency—ArgoCD, automated testing pipelines, and monitoring via Prometheus—treating deployment reliability as a first-class concernFamiliarity with analytical warehousing ...

Senior Databricks Architect Technology: Databricks (Spark, PySpark/SQL) and data lake architectures. Location: London, UK. Compensation: Competitive (including bonus). Job Description We are looking for a Data Architect to design, build, and operate scalable data pipelines and data products, enabling a modern data mesh architecture. The role … focuses on integrating legacy systems with cloud-native platforms to support analytics, reporting, and domain-owned data products. Primary Skills Strong experience with Databricks (Spark, PySpark/SQL) and data lake architectures. Hands‐on experience with streaming technologies, especially Kafka/Confluent Kafka. Experience integrating legacy data sources such ...

scalable, real-time pipelines powering trading decisions, risk management, and market analytics.Key Responsibilities Data Architecture: Build and optimize ETL/ELT pipelines using Databricks, Spark (PySpark/Scala), and Delta Lake for structured/unstructured energy market data.Trading Analytics: Partner with Quants, Risk Managers, and Traders to implement models … Renewables) and trading data types (Reuters, Bloomberg, Trayport, weather, ETRM).Databricks Mastery: Hands-on experience with production workloads on Databricks (Azure/AWS), PySpark, Spark SQL, Delta Lake, Unity Catalog, Python, SQL, orchestration tools (Airflow, Workflows, ADF), and streaming (Kafka/Event Hubs).Professional Skills: Strong consultative mindset, excellent ...

programs. This is a hands-on leadership role for a builder who can equally command a whiteboard in front of a CTO and a Spark UI when a pipeline misbehaves. You will also shape the practice itself — mentoring engineers, creating accelerators and reusable assets, and converting delivery success into … heart — 12+ years in data engineering/architecture, 3+ on Databricks at enterprise scale, and still happiest when hands are on the keyboard. Deep Spark expertise — architecture, performance tuning, streaming, debugging, the advanced stuff that separates architects from diagram-drawers. Lakehouse fluency — Medallion architecture, Delta Lake, Unity Catalog, governance ...