1 to 25 of 48 Apache Spark Jobs in London

urgently require a Data Platform Engineer with core strengths in Building and Operating Data Platforms, and hands on expertise in Azure Services, Synapse Spark Architecture, ADF and Terraform (IaC), to assist in designing and delivering complex cloud solutions. Key Requirements: Proven, hands-on experience of Building and Operating Data … Advanced knowledge of Azure Services Demonstrable experience with Azure Data Factory (ADF), including network configurations and integration runtime options Strong expertise in Synapse Analytics Spark, Azure Key Vault, and Data Lake Storage Accounts Ability to showcase DevOps and CI/CD capabilities Working knowledge of designing and deploying infrastructure ...

Lead, Data Engineering Manager etc. Proven success with modern data infrastructure: distributed systems, batch and streaming pipelines Hands-on knowledge of tools such as Apache Spark, Kafka, Databricks, DBT or similar Experience building, defining, and owning data models, data lakes, and data warehouses Programming proficiency in the likes ...

Excellent communication and documentation skills. · Ability to work independently and lead cross-functional teams. · Adaptability to rapidly evolving technological environments PREFERRED SKILLS · Airflow, dbt, Spark, Kafka, RabbitMQ, Redis. · Git, CI/CD pipelines. · Experience with data warehousing solutions such as Snowflake, Redshift, BigQuery, or Synapse. · Exposure to AI/ ...

Java Commercial experience in client-facing projects is a plus, especially within multi-disciplinary teams Deep knowledge of database technologies: Distributed systems (e.g., Spark, Hadoop, EMR) RDBMS (e.g., SQL Server, Oracle, PostgreSQL, MySQL) NoSQL (e.g., MongoDB, Cassandra, DynamoDB, Neo4j) Solid understanding of software engineering best practices - code reviews, testing ...

data warehousing concepts, and distributed computing. Familiarity with CI/CD, version control, and DevOps practices. Nice-to-Have Experience with streaming technologies (e.g., Spark Structured Streaming, Event Hub, Kafka). Knowledge of MLflow, Unity Catalog, or advanced Databricks features. Exposure to Terraform or other IaC tools. Experience working ...

Data Software Engineer/Developer (Python Spark Azure) London/WFH to £100k Are you a data centric Software Engineer with strong Python coding skills? You could be progressing your career in a senior, hands-on Data Software Engineer role at a scaling, global technical services company as they … collaboratively problem solving as part of an Agile development using a range of technology to create data pipelines with a focus on Python and Spark; you'll be working with Azure, ETL pipelines and CI/CD, ingesting and analysing terabytes of data with varying structures from a range ...

Knowledge of machine learning frameworks (e.g., PySpark, PyTorch) and model deployment tools (e.g., MLflow, TensorFlow Serving). Strong experience with data processing frameworks (e.g., Apache Spark, Flink). Expertise in SQL and NoSQL databases (e.g., MySQL, PostgreSQL, MongoDB, Cassandra). Hands-on experience with cloud platforms (e.g. ...

Quantexa delivery experience Strong knowledge of Entity Resolution, Scoring Framework, Network Generation, and UI configuration 5 -10 years' experience with Big Data technologies (Scala, Spark, Hadoop, Elasticsearch) Solid data modelling and analytics platform design experience Excellent stakeholder communication skills Experience with cloud platforms (AWS, Azure, or GCP) Desirable Experience ...

similarly regulated industry. Strong coding skills in Python or Java (Spring Boot); React experience is a plus. Proficiency with modern data tools: Airflow, Spark, Kafka, dbt, Snowflake or similar. Experience with cloud platforms (AWS, Azure, GCP), containerization (Docker, Kubernetes), and CI/CD. Data Quality: Proven ability to validate ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

Solid understanding of data warehousing, lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

lakehouse and datalake architectures Familiarity with modern data engineering patterns (ETL/ELT, medallion architecture, observability) Excellent Python and SQL skills Experience working with Apache Spark and large-scale data processing For a full consultation on this exciting new role, please get in touch with ARC IT Recruitment ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

technology (AWS or Azure). Strong experience with creation of data ingestion pipeline and ET process. Strong knowledge of big data tool such as Spark, Databricks and Python. Strong understanding of common machine learning techniques and frameworks (e.g. mlflow). Strong knowledge of Natural language processing (NPL) concepts. Strong ...

teams Strong background in cloud data platforms and modern data architectures Hands-on expertise with Python , data modelling, ETL, and big data technologies (e.g. Spark) Experience operating in regulated or complex enterprise environments Excellent stakeholder management and communication skills Comfortable working with ambiguity and driving change Nice to have ...

performance, efficiency, and cost-effectiveness. Implement data quality checks and validation rules within data pipelines. Data Transformation & Processing: Implement complex data transformations using Spark (PySpark or Scala) and other relevant technologies. Develop and maintain data processing logic for cleaning, enriching, and aggregating data. Ensure data consistency and accuracy throughout … data lifecycle. Azure Databricks Implementation: Work extensively with Azure Databricks Unity Catalog, including Delta Lake, Spark SQL, and other relevant services. Implement best practices for Databricks development and deployment. Optimise Databricks workloads for performance and cost. Need to program using the languages such as SQL, Python, R, YAML ...

frameworks. Strong expertise in statistical modelling, predictive analytics, and machine learning. In-depth understanding of data architecture and big data technologies (eg, Hadoop, Spark, AWS). Exceptional problem-solving skills and ability to think strategically. Outstanding communication abilities to convey technical concepts to non-technical stakeholders. Experience in managing ...

coding skills in Java (Spring Boot) for data engineering and services. Proven background working within financial services environments. Hands-on experience building and managing Apache Airflow/Spark and Hadoop clusters. Deep experience in big data engineering, including Hadoop, Kafka, and Spark-based data pipelines. Strong ...

+ 10% bonus + benefits Purpose: Build and maintain large, scalable Data Lakes, processes and pipelines Tech: Python, PySpark, AWS, Iceberg/Kafka, Spark/Glue, CI/CD Industry: Financial services/securities trading Immersum continue to support a leading SaaS securities trading platform, who are hiring … Infra tooling using Terraform, Ansible and Jenkins whilst automating everything with Python Tech (experience in any listed is advantageous) Python Cloud: AWS Lake house: Apache Spark or AWS Glue Cloud Native storage: Iceberg, RDS, RedShift, Kafka IaC: Terraform, Ansible CI/CD: Jenkins, Gitlab Other platforms such ...

engineering and ability to handle large dataset for business insights. Experience with model governance including explainability, observability and documentations Scala and Python knowledge desirable, Apache Spark, Azure platform knowledge is a plus Strong knowledge of SQL Strong Analytical, machine learning and deep learning skills Please send ...

Data Analytics Sales Executive - Data Sales, Talend, Snowflake, Apache Spark, Kafka, Dataops, AWS, Azure, Data visualisation, SQL, ETL, Pre-Sales, Account Manager, SaaS Sales, Cloud Solutions, Digital Transformation, Business Development, Sales Consultant, CXO Engagement, Strategic Accounts, Deal Negotiation, Remote Working, UK-wide, Permanent Role, Sales Targets, New Business ...

MLOps: experiment tracking (e.g., MLflow), model registry, CI/CD for ML. Serving and orchestration: AWS SageMaker, EKS, batch & real-time inference. Data pipelines: Spark/SQL, orchestration (Airflow), monitoring for drift/perf. Nice to Have Cost/performance optimisation on AWS; GPU scheduling; quantisation/distillation. Experience ...

Messaging Technologies with a good understanding of Kubernetes, Dockers and preferably some exposure to GCP (Google Cloud Platform), ReactJS, Python, Big Data Technologies like Spark, Beam Excellent personal organisation, completer/finisher, consensus builder with excellent communication skills and able to lead global development teams both directly ...

extracting, manipulating, and preparing data from relational databases. Experience with NoSQL databases is a plus. Big Data/Cloud: Experience with big data tools ( Spark, Hadoop ) and cloud computing platforms ( AWS, Azure, or GCP ) for scalable ML workflows. Visualization: Ability to create clear, compelling data visualizations using tools like ...

pipelines for performance, efficiency, and cost-effectiveness . Implement data quality checks and validation rules within pipelines. Data Transformation & Processing Implement complex transformations using Spark (PySpark or Scala) and related technologies. Develop logic for data cleaning, enrichment, and aggregation . Ensure data consistency and accuracy throughout the lifecycle. Azure … Databricks Implementation Work extensively with Unity Catalog, Delta Lake, Spark SQL , and other Databricks services. Apply best practices for development and deployment . Optimise workloads for performance and cost . Program in SQL, Python, R, YAML, and JavaScript . Data Integration Integrate data from relational databases, APIs, and streaming ...