1 to 25 of 43 Permanent Apache Spark Jobs in London

British Citizenship, or ILR (Indefinite Leave to Remain), or UK Settlement Visa. We do not provide visa sponsorship. Key Requirements:Must have skills:-Spark & Scala Nice to have skills:-Spark Streaming, Hadoop,-Hive,-SQL,-Sqoop,- Impala Detailed Job Description: At least 8+ years of experience and strong knowledge … fundamental Data Structures and their usage At least 8+ years of experience in designing and developing large scale, distributed data processing pipelines using Apache Spark and related technologies. Having expertise in Spark Core, Spark SQL and Spark Streaming. Experience with Hadoop, HDFS, Hive and other ...

acceptable). Experience with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python. Frameworks/Infrastructure:- Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch ...

Skills & Experience Strong experience with Python and/or SQL in a production environment. Hands-on experience with modern data engineering frameworks such as: Apache Spark, Databricks Airflow/Prefect/Dagster Kafka/EventHub/Kinesis Experience building data solutions on at least one major cloud platform ...

keys, sort keys , and workload management. Implement Redshift Spectrum to query data in S3 using external tables. 3. Data Lake & Iceberg Implement and maintain Apache Iceberg tables on AWS for schema evolution and ACID operations. Build Iceberg-based ingestion and transformation pipelines using Glue, EMR, or Spark. Ensure high … data Billing & revenue assurance Build models and pipelines to support network analytics, customer 360, churn prediction, fraud detection , etc. 7. Performance Optimization & Monitoring Tune Spark/Glue jobs for performance and cost. Monitor Redshift/Athena/S3 efficiency and implement best practices. Perform data quality checks and validation ...

data platform and engineering capability as the business continues to scale its trading operations. They are looking for a Data Engineer with strong Apache Airflow experience to help design and build the pipelines that power trading, analytics and commercial decision-making. This is a great opportunity to join … platform and the opportunity to work with modern open-source technologies . What You’ll Be Doing Designing and building data pipelines orchestrated with Apache Airflow Developing ETL/ELT workflows in Python and SQL to process trading and operational data Building reliable, scalable data infrastructure to support analytics ...

Strong full-stack experience with Python , Java (Spring Boot) , React , SQL, and relational databases. Hands-on experience with modern data engineering tools (e.g. Airflow, Spark, Kafka, dbt, Snowflake or similar). Experience with cloud platforms , containerisation (Docker/Kubernetes) , and CI/CD pipelines . Solid understanding of data ...

GenAI workloads. Strong knowledge of data modelling, ETL processes, SQL, and scripting languages such as Python. Experience with Big Data technologies (e.g., Hadoop, Spark) and data visualisation tools (e.g., Tableau, Power BI). Familiarity with AI/ML frameworks (TensorFlow, PyTorch, Hugging Face, etc.). Experience in implementing security ...

Responsibilities will include; Design, implement, and optimize end-to-end solutions using Fabric components: o Data Factory (pipelines, orchestration) o Data Engineering (Lakehouse, notebooks, Apache Spark) o Data Warehouse (SQL endpoints, schemas, MPP performance tuning) o Real-Time Analytics (KQL databases, event ingestion) o Manage and enhance OneLake ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

Azure, utilizing managed services like SageMaker or Vertex AI. Data Foundations: Strong understanding of data modeling, feature stores, and distributed computing (e.g., Spark, Dask). Qualifications & Experience Experience: 5+ years in a software or data engineering role, with at least 3 years specifically focused on the deployment and scaling ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

extracting, manipulating, and preparing data from relational databases. Experience with NoSQL databases is a plus. Big Data/Cloud: Experience with big data tools ( Spark, Hadoop ) and cloud computing platforms ( AWS, Azure, or GCP ) for scalable ML workflows. Visualization: Ability to create clear, compelling data visualizations using tools like ...

DevOps and broader software engineering standards. Cloud experience (any platform). Previous mentoring experience. Nice to have: Snowflake or Databricks Spark, PySpark, Hadoop or similar big data tooling BI exposure (PowerBI, Tableau, etc.) Interview Process The process is fully structured, transparent, and efficient: Video call - high-level overview ...

uplift practices and delivery maturity. Required Technical Skills & Expertise AWS Data & Analytics (Hands-On Expertise) AWS Glue (ETL/ELT development, PySpark, Glue Catalog) Apache Spark AWS Redshift (modelling, optimisation) AWS Lambda AWS RDS AWS Step Functions Experience with Open Table Formats: Delta Lake Apache Hudi Strong ...

security controls. Monitor, maintain and optimise data platforms to ensure performance and reliability. Skills and Experience Required: ETL development using Python or similar languages Apache Spark, NiFi, or Kafka Relational (PostgreSQL, MySQL) and NoSQL (MongoDB, Cassandra) databases AWS and cloud-based data services Palantir Foundry Stakeholder Engagement ...

integration and API development (REST, SOAP) Informatica or Oracle Data Integrator SQL Server Integration Services (SSIS) Azure Data Factory or Azure Synapse Databricks or Apache Spark Amazon Redshift SQL Server or Oracle Database Oracle Integration Cloud or Business Objects Data Services Desirable Skills: Business intelligence tools such ...

preparation. Machine learning and statistical methods, including model validation. Data analytics and visualisation techniques. Processing large datasets using batch or streaming tools such as Apache Spark. Data acquisition and fusion approaches. Cloud platforms (ideally AWS) and cloud-based data science solutions. DataOps principles and practices. Structured and unstructured databases. ...

models, transformers, retrieval models). Hands-on experience with LLMs and GenAI technologies. Strong programming and problem-solving skills with proficiency in Python, SQL, Spark, and Hive. Deep understanding of classical and modern ML techniques, A/B testing methodologies, and experiment design. Solid background in ranking, recommendation ...

from classical statistics and regression to deep learning and NLP. Data Engineering: Proficiency in SQL and experience working with modern data stacks (e.g., Snowflake, Spark, or cloud-native tools on AWS/GCP/Azure). Scientific Rigor: A strong foundation in the scientific methodforming hypotheses, designing experiments ...

processes Our ideal candidate has: Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms ...

years of Scala experience Experience delivering and maintaining distributed data processing systems using Big Data tools, lakehouse platforms, and workflow orchestration (experience with Spark, Databricks, or Airflow is a plus) Experience building with cloud platforms such as AWS, Azure, or GCP Experience working with AI tools or platforms (LLMs ...

Data Science Strong Python skills and experience with cloud platforms (AWS, GCP, or Azure) Experience building and deploying production ML systems Familiarity with Databricks, Spark, or MLOps workflows (nice to have but not essential) Solid understanding of modern ML methods and data engineering fundamentals Strong academic background (ideally Russell ...

data science or statistical role. Expertise in advanced analytics techniques including machine learning, optimisation and segmentation. Strong proficiency in Python, SQL and Spark SQL. Experience working with large datasets and distributed computing tools such as Azure Synapse and Snowflake. Familiarity with data transformation tools (Databricks, Azure Data Factory ...

/CD , Git-based workflows, and automation Strong problem-solving skills and a collaborative mindset Bonus Terraform or other IaC tools Slurm, Kueue, Ray, Spark, or similar systems GPU tooling (CUDA, Nvidia operators, schedulers) Experience supporting ML training or data science teams ...

RNNs, Transformers) and when to apply them Strong SQL and data manipulation capabilities Experience working with large-scale datasets and distributed compute frameworks (e.g., Spark) Demonstrated ability to independently lead technical ML solution design Experience working in client-facing delivery environments Exposure to cloud-based ML platforms (AWS, Azure ...