51 to 75 of 254 Apache Spark Jobs in London

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...

Must-Have Skills & Experience Deep expertise designing and building scalable data pipelines and data products in cloud environments Strong hands-on experience with Databricks, Spark, SQL, and Python Experience developing and optimizing batch and real-time data processing solutions Strong understanding of modern data architecture patterns, including Kimball, Data ...

BigQuery, Dataflow, Dataproc, Cloud Composer (Airflow), Pub/Sub, Cloud Storage, Bigtable, and Cloud SQL. Proficiency in Python and SQL, with experience in Spark or Apache Beam. Experience designing data warehouse and data lake architectures, including data modeling. Working experience with Terraform and CI/CD tooling such ...

working with cloud‐native technologies, and applying robust software engineering practices. Key Responsibilities Design, develop, and optimise scalable, testable data pipelines using Python and Apache Spark. Implement batch workflows and ETL processes adhering to modern engineering standards. Develop Cloud-Based Workflows Orchestrate data workflows using AWS services such … Glue, EMR Serverless, Lambda, and S3. Contribute to the evolution of lakehouse architecture leveraging Apache Iceberg. Apply Software Engineering Best Practices Use version control, CI/CD pipelines, automated testing, and modular code principles. Participate in pair programming, code reviews, and architectural design sessions. Data Quality & Observability Build monitoring ...

database design principles Experience with Microsoft Azure and Azure Data Lake Storage (ADLS) Experience with data engineering tools such as Azure Data Factory, Databricks, Apache Spark, dbt or Airflow Experience with cloud data platforms such as Azure Synapse Analytics or Snowflake Experience using Git for version control Experience ...

taking customers from EDW to the full capabilities of the Data Intelligence Platform. Demonstrated competence in the Lakehouse architecture including hands-on experience with Apache Spark, Python and SQL Excellent communication skills; both written and verbal Public Cloud Architecting: Hands-on experience designing and implementing big data architectures ...

environments Owning the cloud infrastructure underpinning data systems through a DevOps approach Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop Good understanding of the possible architectures involved in modern data system design (e.g. Data Warehouse, Data Lakes and Data ...

grade code Distributed Data Processing Experience building and operating: batch processing systems distributed data systems near real-time streaming pipelines Familiarity with: Kafka Flink Spark Databricks experience is a strong plus Databases & Data Warehousing Comfortable working with: RDBMS data warehousing solutions Strong understanding of data modelling aligned with business ...

Pulsar, Kafka, RedPanda. Must have knowledge and experience automatically provisioning and supporting multiple big data technologies; such as MongoDB, DataBricks, Big Query, Hadoop, Spark, Trino, DataLakes/Lakehouse Architectures. Infrastructure management with Terraform Experience with Github or any other source control Knowledge of Scripting/Coding languages - Go, Bash ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI). Data Science: Familiarity or experience with classical NLP techniques (BERT, topic ...

services, a good understanding of cloud architecture, and proficiency with Terraform for provisioning and managing cloud infrastructure Experience building streaming data pipelines using Apache Flink, Apache Kafka, Spark Structured Streaming, or similar technologies Knowledge of modern data lakehouse architectures and ecosystem such as Kafka, Flink, Spark … with a SaaS or FinTech company Technologies we use: AWS Services: MSK, EC2, VPC, IAM, S3, SQS, Managed Flink, EKS Data Lakehouse: Kafka, Flink, Spark, Databricks, DBT, Debezium #J-18808-Ljbffr ...

cleansed, conformed business layers) while securely handling identifiable data. Scalable Data Processing: Develop, tune, and monitor large-scale batch processing pipelines using Azure Databricks (Spark-based processing) and PySpark. Orchestration & Workflow: Build automated data workflows using Microsoft Fabric Data Factory and Azure Data Factory to move data efficiently across … Microsoft Azure, with practical exposure to the Microsoft Fabric ecosystem (OneLake, Data Factory, Fabric Lakehouse). Big Data Processing: Deep technical expertise with Apache Spark/Azure Databricks for distributed compute, memory optimization, and Delta Lake formats. Data Engineering Languages: Advanced proficiency in SQL (T-SQL, SparkSQL ...

training and evaluation. Ideal candidates have strong SQL and Python skills, experience with ETL/ELT, orchestration, streaming and data warehousing, and familiarity with Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. #J-18808-Ljbffr ...

Designed and delivered modern data platforms end to end on AWS and/or Databricks Still writes production-grade code: strong Python and SQL, Spark, dbt or equivalent, infrastructure-as-code (Terraform preferred) Track record in a consultancy, systems integrator or other client-facing setting comfortable being accountable ...

Designed and delivered modern data platforms end to end on AWS and/or Databricks Still writes production-grade code: strong Python and SQL, Spark, dbt or equivalent, infrastructure-as-code (Terraform preferred) Track record in a consultancy, systems integrator or other client-facing setting — comfortable being accountable ...

culture where people feel safe to challenge ideas, learn, and grow. Nice to have Hands-on familiarity with the modern data stack — for example, Spark, Flink, Kafka, Debezium, Airflow, dbt, or similar — and with cloud platforms (AWS/GCP or equivalent) and modern DevOps practices. Experience leading teams that ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

into a coherent operational story. Data engineering & processing systems: You have built, operated, or supported production data pipelines using distributed processing systems such as Apache Spark or Databricks, and understand common failure modes, performance tradeoffs, and operational challenges in batch and hybrid pipelines. Data platforms & analytics systems ...

Collaborate with cross-functional teams to align data architecture with financial regulations (e.g., GDPR, Basel III, MiFID II). Drive adoption of Delta Lake , Apache Spark , and MLflow for advanced analytics and machine learning use cases. Provide technical leadership and mentoring to data engineers and analysts across financial ...

looking for a core stack of SQL and Python, with experience across ETL and ELT, orchestration, streaming and data warehousing, using tools such as Spark, Kafka, Airflow, dbt, Snowflake, BigQuery or Databricks on AWS, GCP or Azure. Some roles focus on hands-on delivery, others on documenting and structuring ...

concept to production. Strong knowledge of Python and core ML libraries (e.g., spaCy, NumPy, SciPy, Transformers). Understanding of big data and cloud technologies (Spark, Hadoop, Azure, AWS, or GCP). Deep expertise in MLOps principles, scalable model deployment, and model governance. Excellent communication and stakeholder management skills with ...

databases, e.g. SQL Server, PostgreSQL, Oracle, MySQL; Data visualisation software: Spotfire, Tableau, or Power BI; Azure cloud computing platform; Big data technologies such as Spark, Elasticsearch, Hadoop; Statistical techniques (regression, clustering etc.); Machine learning and pattern recognition; Front-end web development e.g. HTML, JavaScript; We are looking for tenacious ...

alerting for ML systems (model drift, data quality, feature coverage, etc.) Design and scale data ingestion and feature transformation flows using batch (e.g., Spark/BigQuery) and streaming (Kafka or equivalent) Contribute to internal Python libraries and platform tooling that accelerate experimentation and deployment for all model teams Ensure ...

modern cloud engineering practices. Wherever you land, you’ll be working with some of the biggest datasets in the UK - using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Cloud-native AI engineering and large-scale data platforms ...

plans and resource requirements accordingly. High proficiency in Python and SQL. Understanding of Snowflake's machine learning capabilities and integration with tools like Python, Spark, or DataRobot. Strong understanding of data governance, privacy, and security principles including IG and GDPR. Familiarity with version control tools (e.g. GitHub or Azure ...