26 to 50 of 254 Apache Spark Jobs in London

delivery across a range of projects, including data analysis, extraction, transformation, and loading, data intelligence, data security and proven experience in their technologies (e.g. Spark, cloud-based ETL services, Python, Kafka, SQL, Airflow) You have experience in assessing the relevant data quality issues based on data sources & uses cases ...

environments. Technical Skills: Experience building and operating ETL/ELT pipelines. Hands-on experience with modern data stacks — our tech includes Python, SQL, Snowflake, Apache Iceberg, AWS S3, PostgresDB, Airflow, dbt, and Apache Spark, deployed via AWS, Docker, and Terraform (experience with some of these or similar ...

agreed scope, acting as the key point of escalation for decisions that impact your area of work - Demonstrable experience of building data pipelines using Spark, Python or SQL - Experience working with AWS and its data and AI/ML services such as SageMaker - Experience of big data and lakehouse ...

working with or within the public sector Familiarity using AI tools to enhance the quality & pace of your output Technologies we love Languages: Python, Spark, SQL Tools: AWS, Azure, Git, Rest APIs, Kubernetes, Docker, Terraform, Datadog Datastores: Databricks, OpenSearch, Postgres Equal Opportunity Statement At Altana, we believe that ...

deliver scalable data architectures across AWS and Azure environments. Define and implement automated data pipelines using cloud-native technologies and frameworks such as Apache Spark. Design data integration solutions connecting cloud platforms with on-premises data sources. Establish event-driven architectures to support efficient data processing, workflow orchestration … Azure. Strong understanding of cloud data architecture, data integration and distributed data processing. Experience designing and implementing automated ETL/ELT pipelines. Experience with Apache Spark or equivalent distributed data processing technologies. Experience designing event-driven architectures and data processing workflows. Strong understanding of data pipeline orchestration, performance ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like ...

learning or data science.* Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials.* Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives).* Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow ...

experience with one or more cloud data ecosystems: Azure (Data Factory, Synapse, Fabric, ADLS, Databricks) AWS (Glue, Redshift, EMR) GCP (BigQuery, Dataflow) Databricks (Spark/PySpark) Snowflake (SQL, Snowpark, performance tuning) Strong SQL and/or Python capability, and familiarity with tools such as dbt, Git, CI/ ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

Company description Publicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom and Zenith, as well as global accelerator PMX; immersive experience group PMCI and access to integrated platform-based technologies and offerings from CJ, Epsilon … marketing platforms. Build and optimize schemas, transformations, and ETL logic for high‐performance, reusable data workflows. Contribute directly to the DFP codebase (Python, SQL, Spark) — developing pipelines and infrastructure alongside the engineering team. Establish data versioning, testing, and governance frameworks ensuring reliability, lineage, and compliance. Data Engineering Leadership Drive ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

with data warehousing concepts and technologies (e.g., Azure Synapse Analytics, Azure Data Lake Storage). Familiarity with big data technologies and frameworks (e.g., Hadoop, Spark) is a plus. Knowledge of data governance and data security best practices. Strong problem-solving skills and attention to detail. Excellent communication and collaboration ...

Azuretechnologies, including ADF, ADLS, Databricks, Stream Analytics, DeltaLive Tables, and Synapse Analytics. Advanced proficiency in SQL andprogramming languages such as Python, Scala, or Spark-SQL. Experience in designing anddeveloping scalable ETL pipelines and optimizing data workflows. Deep understanding of data migrationfrom on-premises to cloud and cloud to cloud. ...

maintain SC clearance. Desirable AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...

Company Description Publicis Media harnesses the power of modern media through leading agency brandsDysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, StarcomandZenith, as well as global acceleratorPMX; immersive experience group PMCI and access to integrated platform-based technologies and offerings fromCJ, EpsilonandInfluential.A key business solution of Publicis ...

large-scale ETL development. Proficiency in SQL and programming languages such as Python, Java, or Scala. Hands‐on experience with big data frameworks (e.g., Spark, Hadoop, Flink) and cloud platforms (AWS, GCP, or Azure). Familiarity with data warehouse technologies (e.g., Snowflake, Redshift, BigQuery) and workflow orchestration tools (e.g. ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...

including lakehouses, data warehousing, ETL/ELT pipelines, orchestration, and dimensional data modelling. Hands‐on experience with modern data technologies such as dbt, Databricks, Spark, SQL, and Python. Understanding of semantic layers, data governance, and the foundations required for trustworthy AI‐powered analytics. Strong technical foundation, with experience designing ...

Strong hands-on experience with Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise ...

Strong hands-on experience with Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise ...

intellectually rewarding, involving large datasets, performance-critical pipelines, and precision engineeringWe value strong individual contributors — this is a hands-on role where your Scala Spark expertise will directly shape the platform's futureWe are a team that is enthusiastic about what we are building — there is a genuine drive … have the opportunity to contribute to architectural decisions, not just execute themRole Overview/What will you do:Build and optimise large-scale Scala Spark data pipelines on Databricks, focusing on the reconciliation domain — writing clean, modular, and production-ready codeDrive modernisation efforts by refactoring legacy Hadoop-based pipelines ...

BigQuery, Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such ...

working with cloud‐native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such … Glue, EMR Serverless, Lambda, and S3. Contribute to the evolution of lakehouse architecture leveraging Apache Iceberg. Apply Software Engineering Best Practices Use version control, CI/CD pipelines, automated testing, and modular code principles. Participate in pair programming, code reviews, and architectural design sessions. Data Quality & Observability Build monitoring ...

product lifecycle. Hands‐on experience with large‐scale data processing and machine learning in production environments. Strong expertise in distributed computing frameworks such as Apache Spark or similar technologies. Solid understanding of machine learning workflows, including model development, deployment, and inference. Strong knowledge of modern data architectures, including ...

taking customers from EDW to the full capabilities of the Data Intelligence Platform. Demonstrated competence in the Lakehouse architecture including hands-on experience with Apache Spark, Python and SQL Excellent communication skills; both written and verbal Public Cloud Architecting: Hands-on experience designing and implementing big data architectures ...