51 to 75 of 409 Apache Spark Jobs in London

complex stakeholder environments. Strong understanding of data pipelines, ETL/ELT frameworks, orchestration tooling, and data engineering best practices. Familiarity with technologies such as Spark, Python, SQL, Delta Lake, Unity Catalog, dbt, Azure Data Factory, Airflow, Kafka, or equivalent tooling. Relevant certifications such as Snowflake SnowPro, Databricks Certified Data ...

modern data architectures, including cloud-based and distributed systems. Strong proficiency in SQL and Python. Experience with data engineering and pipeline technologies such as Apache Airflow and Spark. Experience working with cloud platforms, particularly AWS and Azure. Knowledge of big data technologies and distributed data processing. Understanding of data ...

comfortable operating across both clouds. Familiarity with OLAP and OLTP storage systems and an eagerness to build expertise in stream processing technologies such as Apache Flink, Kafka, or Spark. (Nice to have: ClickHouse.) Familiarity with observability tooling such as Grafana or Datadog, and a good instinct for keeping systems ...

experience with one or more cloud data ecosystems: Azure (Data Factory, Synapse, Fabric, ADLS, Databricks)AWS (Glue, Redshift, EMR)GCP (BigQuery, Dataflow)Databricks (Spark/PySpark)Snowflake (SQL, Snowpark, performance tuning)Desirable ExperienceStrong SQL and/or Python capability, and familiarity with tools such as dbt, Git, CI/ ...

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy click apply ...

maintain scalable Databricks pipelines (ETL, streaming, ML workflows). Perform cluster and job performance tuning: optimize cluster sizing, caching, partitioning, and shuffle management. Monitor Spark job metrics, analyze logs, and identify bottlenecks in data throughput or latency. Implement cost-optimization strategies for Databricks jobs and clusters using autoscaling … Databricks REST APIs, Prometheus, or Azure Monitor. Maintain best practices for data engineering performance, testing, and documentation. Technical Skills Required Strong proficiency with Databricks, Apache Spark (SQL, PySpark, Scala), and Delta Lake. Deep understanding of Spark execution plans, shuffle optimization, and data partitioning strategies. Experience tuning ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

value. Excellent analytical, problem-solving, and communication skills. Proven experience leading data modernisation initiatives across multiple sectors. Technical Expertise Proficiency with big data technologies: Apache Spark, Kafka, Confluent, Databricks, Unity Catalog. Strong knowledge of cloud ecosystems (AWS, Azure, GCP) and infrastructure-as-code (Terraform, CloudFormation). Hands ...

Azure, GCP).Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon).Familiarity with data lake architectures and distributed processing frameworks (e.g., Spark, Hadoop).Experience with version control tools (GitHub, Bitbucket) and CI/CD pipelines. Understanding of data governance, security, and compliance frameworks. Exposure to AI/ ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

deliver agreed scope, acting as the key point of escalation for decisions that impact your area of workDemonstrable experience of building data pipelines using Spark, Python or SQLExperience working with AWS and its data and AI/ML services such as SageMakerExperience of big data and lakehouse platforms ...

working with or within the public sector Familiarity using AI tools to enhance the quality & pace of your output Technologies we love Languages: Python, Spark, SQL Tools: AWS, Azure, Git, Rest APIs, Kubernetes, Docker, Terraform, Datadog Datastores: Databricks, OpenSearch, Postgres Equal Opportunity Statement At Altana, we believe that ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/ ...

Security In-depth knowledge of the financial services industry and their IT systems Practical cloud native experience Preferred qualifications, capabilities, and skills Advanced Apache Spark experience(PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. Deep ...

/CD, Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

Designing scalable batch and streaming data architectures , ensuring performance, reliability and maintainability. Establishing architectural standards, reusable patterns and engineering frameworks. Providing technical guidance across Spark, PySpark, AWS Glue, Airflow and associated technologies . Supporting the architecture and development of new data products. Collaborating with engineering teams to translate architectural … large-scale enterprise data architectures . Deep expertise across the AWS data ecosystem , including AWS Glue and associated data services. Strong technical understanding of Apache Spark, PySpark and Apache Airflow . Proven experience designing complex batch and streaming data processing solutions . Experience supporting major data transformation ...

throughput batch and streaming data pipelines using Kafka, Flink, and ETL technologiesDesign and build unified analytics engine designed for processing large-scale data using Apache Spark and related toolsDevelop and optimize analytical data models for time-series, financial metrics, and trading activityImplement and manage analytical databases (ClickHouse, MongoDB ...

with data warehousing concepts and technologies (e.g., Azure Synapse Analytics, Azure Data Lake Storage). Familiarity with big data technologies and frameworks (e.g., Hadoop, Spark) is a plus. Knowledge of data governance and data security best practices. Strong problem-solving skills and attention to detail. Excellent communication and collaboration ...

large-scale ETL development. Proficiency in SQL and programming languages such as Python, Java, or Scala. Hands‐on experience with big data frameworks (e.g., Spark, Hadoop, Flink) and cloud platforms (AWS, GCP, or Azure). Familiarity with data warehouse technologies (e.g., Snowflake, Redshift, BigQuery) and workflow orchestration tools (e.g. ...

maintain SC clearance. AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...

maintain SC clearance. Desirable AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...

following: Python, SQL, JavaCommercial experience in client-facing projects is a plus, especially within multi-disciplinary teamsDeep knowledge of database technologies: Distributed systems (e.g., Spark, Hadoop, EMR)RDBMS (e.g., SQL Server, Oracle, PostgreSQL, MySQL)NoSQL (e.g., MongoDB, Cassandra, DynamoDB, Neo4j)Solid understanding of software engineering best practices – code reviews ...