226 to 250 of 254 Apache Spark Jobs in London

observability, and end-to-end platform quality. You will mentor engineers, partner with cross-functional teams, and help evolve the lakehouse architecture using Hive, Spark, and Iceberg. This role offers high-impact work at scale in a fast-paced environment. #J-18808-Ljbffr ...

Hyperledger framework would be very appreciated A first experience using Mobile technologies in Android/iOS environment, Big Data technologies such as Cassandra/Spark/NoSQL is an asset Work within our team of passionate and talented developers Design, implement and test platforms and Apps Deal with cryptocurrencies ...

building data platform capabilities that support data and machine learning workloads across ASOS. Developing high-performance data pipelines and processing frameworks using Python, Scala, Spark and Databricks. Owning and evolving platform components that help engineers and data scientists build, test, deploy and monitor data products. Improving platform reliability, observability … likely to have: Experience building or operating large-scale data platforms or data-intensive applications in a cloud environment. Experience working with Databricks, Spark and distributed data processing technologies. Experience designing data architectures that balance scalability, reliability and cost efficiency. Experience implementing modern engineering practices, including CI/ ...

equivalent platforms, with the ability to partner with technology teams to shape architecture and drive enterprise adoption. Proficiency across Python, q/KDB, SQL, Spark, C#, and related engineering toolsets, with sufficient depth to guide technical decisions, challenge design options, and work effectively with engineering teams to deliver scalable … Science, or related discipline. Nice to Have Experience integrating real-time market data into Align, Endur, and Front Office trading platforms. Familiarity with Flink, Spark Streaming, and modern stream-processing technologies. Knowledge of execution algorithms, quantitative trading strategies, and best execution frameworks. Experience with MDSL, TRG Screen, DACS ...

needs of the department. This is an exciting opportunity to work on cutting edge technologies including Big Data and NoSQL databases (Hadoop, HBase, Hive, Spark and MongoDB) to allow the business to gain advanced insight into their portfolios and valuation metrics. Key responsibilities include:* Lead business requirements elicitation, analysis … documentation efforts* Assist technology team in Azure cloud solution implementation; write Spark SQL queries to analyze and transform data; provide feedback to User Interface team; testing and documentation* Analyze solution requirements; understand project scope; determine project priorities; ensure efficient and on-time delivery of project tasks and milestones; follow ...

strategy and promote best practices amongst engineers and data scientists. How to make an Impact Architect and Implement : Own the technical roadmap for our Spark-based data processes to ensure our Airflow pipelines are performant, cost-effective, and scalable. You will enhance our existing services through hands-on development … deployment. Expert-level Python and its data ecosystem (Numpy, Pandas), including designing frameworks for data tasks. Deep understanding of distributed data processing engines like Apache Spark. Ability to reason about the full data lifecycle from ingestion through to real-time serving - your system designs will need to meet downstream ...

Strong written and verbal communication skills, including the ability to explain complex methodology to diverse audiences. You might also have Familiarity with Databricks, DBT, Spark/PySpark, or similar large‐scale data platforms. Modified infrastructure‐as‐code (e.g. Terraform) or contributing to production data pipelines. Mentored junior data scientists … engineers. Knowledge of, or passion for, the cryptocurrency and blockchain space. Technologies we use Python SQL Databricks DBT PySpark/Spark Terraform AWS Postgres We encourage applicants across any race, ethnicity, gender/gender expression, age, spirituality, ability, experience and more. If you need any accommodations to make ...

operating large‐scale distributed platforms that power our research, trading and engineering teams across on‐premises and AWS environments, using technologies such as Spark, Trino, Kafka, ClickHouse and Airflow. Key Responsibilities Building, operating and scaling distributed analytics platforms across on‐premises and AWS environments Designing and implementing new platform … hours rota to support critical systems Core Skills and Technologies Experience running distributed data and analytics systems at scale using tools such as Spark, Kafka, Trino or Airflow Strong Linux skills and proficiency in Python for automation and integration Familiarity with infrastructure as code, using Terraform or Ansible Deep ...

engineering teams. You’ll deliver highly available, secure and high-performance analytics services across on-premises and AWS environments, using technologies such as Spark, Trino, Kafka, ClickHouse and Airflow. This is a highly collaborative position, working directly with researchers and engineers to evolve platform capabilities, automate operations and explore … critical systems Who are we looking for? Core skills and technologies Experience running distributed data and analytics systems at scale using tools such as Spark, Kafka, Trino or Airflow Strong Linux skills and proficiency in Python for automation and integration Familiarity with infrastructure as code, using Terraform or Ansible ...

applicationswith enterprise systems. Develop retrieval-augmentedgeneration (RAG) pipelines, including document ingestion, embeddings,indexing and retrieval. Build and maintain dataprocessing pipelines using Databricks and Apache Spark. Support the development,testing and deployment of machine learning models through MLOps pipelines. Develop user-facing AIinterfaces, including streaming interactions and human … Generative AI, Large LanguageModels (LLMs), RAG, prompt engineering and AI agent frameworks . Experience working with largedatasets and data processing technologies such as Apache Spark. Strong understanding ofsoftware testing, CI/CD, application security and operational resilience. Practical experience withcloud-native technologies, containers and Kubernetes. Experience working within Agiledevelopment ...

Scala), with a track record of building and maintaining reliable, production-grade software at scale. Hands-on experience with data transformation tools such as Spark and DBT, with a focus on building well-tested, maintainable data products is a strong advantage. Confidence in working with Kubernetes-native platforms … deploy and operate data workloads across cloud environments is preferred. Knowledge of Golang would be an advantage. Familiarity with Apache Iceberg table format and schema management via catalogs such as AWS Glue, with an appreciation for data quality and consistency is a strong advantage. Solid grounding in core ...

schema changes Troubleshoot data-quality and pipeline issues Implement data-quality checks, testing and documentation Work with distributed data technologies such as Presto, Spark or Hive Use workflow orchestration tools such as Airflow Participate in code reviews and Git-based development workflows Collaborate with Data Engineers, Analysts and Engineering … 5+ years' experience building production data pipelines Advanced SQL , including performance optimisation on large tables Strong Python for data engineering Experience with Presto, Spark, Hive or similar Experience with Airflow or another workflow scheduler Strong ETL/data pipeline development experience Git and code-review experience Strong understanding ...

project objectives and timelines. Hands-on Kubernetes experience Hands-on AWS experience Databases and middleware — Postgres, Kafka, Redis Big data stack — data warehousing, MPP, Spark, Flink Micro-service architecture and service mesh UI development experience What You Will Bring 3+ years of software development experience Strong command … Jenkins or CodeDeploy Preferred Hands-on Kubernetes experience Hands-on AWS experience Databases and middleware — Postgres, Kafka, Redis Big data stack — data warehousing, MPP, Spark, Flink Micro-service architecture and service mesh UI development experience Benefits Full time benefits for FT #J-18808-Ljbffr ...

Scala programming to develop robust, scalable, and high-performance data processing applications. Build and optimise large-scale data processing and analytics pipelines using Apache Spark. Develop and maintain end-to-end data pipelines for both batch and real-time data processing. Apply solid understanding of data engineering concepts, including … business stakeholders. Skills & Experience: Proven experience as a Software Engineer with a strong focus on Scala development. Extensive hands-on experience with OpenShift and Apache Spark. In-depth understanding of data engineering principles and practises. Experience in developing and maintaining scalable data pipelines for batch and real-time processing. ...

interest or experience in data/network security would be a nice-to-have Datadog/Grafana/Prometheus Data related products (Airflow, Jupyter, Spark, dbt etc) The projects will be varied and we're looking for someone who can work autonomously and proactively to scope problems and solve … tracking CircleCI for continuous deployment Parquet and Delta file formats on S3 for data lake storage Postgres/Aurora for our relational databases Spark for data processing dbt for data modelling SparkSQL for analytics Streamlit for data applications Are you ready for a career with us? We want ...

hear from you! Advanced Stream Processing & Event-Driven Architecture: Proven track record building, deploying, and operating production‐grade, low-latency stream processing pipelines using Apache Kafka (or Kafka Streams, Flink, Spark Streaming). Idempotency & Event Reliability: Deep hands‐on experience handling duplicate events, out‐of‐order log processing ...

workflows, applications, transformations, and reusable components. Support the migration of data assets, ETL pipelines, and workloads from the Cloudera Hadoop platform (HDFS, Hive, Impala, Spark) to AWS Cloud. Design, develop, and optimise data ingestion, transformation, and loading processes into Amazon S3 Data Lake and Amazon Redshift. Develop and maintain … based data integration, transformation, validation, and reconciliation processes. Integrate and support data services including AWS Glue Data Catalog, Athena, Redshift Spectrum, EMR Spark Perform data migration, testing, reconciliation, and unit testing activities to ensure data integrity and successful platform migration. Create and maintain Unix/Linux shell scripts supporting ...

with our Platform Engineering team, you'll own data pipelines from ingestion through transformation, governance and consumption. You'll work with technologies including Databricks, Spark and Delta Lake to make complex data reliable, secure and ready for use across OQC's customer-facing applications. You’ll collaborate across Software … partnering across engineering teams to deliver reliable data foundations for customer-facing products. What We're Looking For Strong hands-on experience with Databricks, Spark, Delta Lake and SQL. Experience building and operating streaming and batch data pipelines. Strong understanding of data quality, lineage, access control and dataset governance. ...

pipelines for large‐scale cybersecurity analytics Productionise and optimise pipelines with a focus on cloud cost optimisation and data asset curation Use Databricks/Spark and advanced analytics to deliver actionable insights Cloud‐Native Analytics & AI/ML Enablement Design, deliver and maintain cloud‐native ML/analytics pipelines … excellent coding ability (repositories and technical reviews) Experience building and maintaining scalable cloud‐based applications ( Azure preferred ) Experience with big data technologies (e.g., Databricks, Spark ) Ability to orchestrate and deliver complex change across teams and geographies Strong stakeholder management and the ability to explain complex engineering topics ...

Tripledot Studios is one of the largest independent mobile games companies in the world. We are a multi-award-winning organisation, with a global 2,500+ strong team across 12 studios. Our expanded portfolio includes ...

modeling and the creation of type-safe and easily testable batch jobs. The team also orchestrates a compute environment that simplifies the execution of Spark jobs, facilitating the analysis of terabytes of data with ease and cost-effectiveness. The runtime environment is based on Hadoop clusters on EC2 spot … Identifying and implementing computational improvements Making improvements and/or fixing bugs by submitting patches to open-source projects in our stack, such as Spark, Iceberg, Hadoop, Airflow, and many others Acting as an expert on Spark and Scala Mentoring engineers (including through pairing, design discussions, and knowledge ...

fundamentals: concurrency, memory management, networking and performance optimisation. In-depth understanding of at least one distributed or big-data system, such as Kafka, Flink, Spark, Trino, Iceberg or Cassandra. Experience gathering requirements and turning ambiguous business needs into clear, accurate technical specifications. A focus on quality and craftsmanship … reason well about complex failure modes and deliver well-tested, maintainable code. Preferred Qualifications Practical experience maintaining large-scale data pipelines with Spark and Flink. Experience with cloud, container and infrastructure tools such as Kubernetes and Docker. Effective use of AI tools such as Claude ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn … scalability, reliability, and observability.* Strong knowledge of the JVM and server-side Kotlin/Java programming.* Solid understanding of the Hadoop ecosystem, including Spark, Hadoop, and Hive.* Strong experience with cloud platforms and data infrastructure, especially AWS, including EMR, S3, and Glue.* Experience working in Agile environments and contributing ...

replication tool, and Waggle Dance, a federation service that allows access to data lake tables across multiple catalogs. Recently, we added Hive support to Apache Iceberg, a high-performance open table format for large analytic datasets that underpins many modern Lakehouse architectures. We guarantee you’ll learn … scalability, reliability, and observability. Strong knowledge of the JVM and server-side Kotlin/Java programming. Solid understanding of the Hadoop ecosystem, including Spark, Hadoop, and Hive. Strong experience with cloud platforms and data infrastructure, especially AWS, including EMR, S3, and Glue. Experience working in Agile environments and contributing ...

Models (LLMs), alongside sub‐agents, tools and MCP integrations . Architect end‐to‐end ML pipelines, integrating large‐scale data processing technologies such as Spark and Airflow with model training, evaluation, deployment and monitoring workflows. Define and evolve experimentation strategies, including A/B testing, offline evaluation and online … building production AI agents using modern agentic architectures and tooling. You have proven experience designing scalable data and ML pipelines using technologies such as Spark, Airflow , or comparable distributed data and orchestration systems. You have demonstrated the ability to operate as a senior individual contributor, setting technical direction ...