1 to 25 of 63 Apache Spark Jobs in London

making AI tools a core part of how you work. Experience with any part of our data stack would be beneficial: Django REST Framework, Apache Airflow, Apache Spark, Apache Iceberg, Apache Kafka, Terraform, and AWS (EMR, ECS). We manage much of this ourselves … experience running your own Airflow cluster or working with Spark at the infrastructure level rather than just the API level is particularly valuable. The Company At Modo Energy, we're on a mission to build the information architecture for the energy transition - we want to be the only place ...

capabilities across the organization. Key Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud … stakeholders to deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

acceptable). Experience with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python. Frameworks/Infrastructure:- Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch ...

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across Cloud (Azure, AWS, GCP) and Big Data (Spark, Kafka) stacks. Collaboration: Work closely with business teams to define data flows and ensure security protocols (encryption/authentication) are met. Technical Toolkit Languages ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

analytics environments. Knowledge of additional AWS services such as S3, Lambda, Athena, Kinesis, EMR, Fargate, or API Gateway. Experience with big data technologies including Spark, Databricks, Hadoop, or Dask. Familiarity with data visualisation tooling such as Plotly. Experience mentoring junior engineers or leading technical initiatives. What You’ll Receive ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

quant teams Cloud experience (AWS preferred: S3, Lambda, Athena, EMR, etc.) CI/CD and Infrastructure-as-Code (CDK, CloudFormation) Big data tools (Spark, Databricks, Hadoop, Parquet, etc.) API development and data integration (REST, web APIs) Interest in AI/ML systems and modern data tooling Experience mentoring ...

English and Maths are also required. Familiarity, and some experience of the following would be beneficial: Proficiency in SQL , including Spark SQL and MS SQL Server , for querying, data manipulation, and performance optimization. Hands-on experience with Python and PySpark for data processing, transformation, and automation tasks. Proven ability ...

models are always running, always accurate, and always improving. Key Responsibilities ML Pipeline Orchestration & Automation Own and extend Circadia’s ML pipeline orchestration using Apache Airflow, including training, evaluation, and deployment workflows. Build and maintain automated pipelines for model retraining, validation, and promotion across development, staging, and production environments. … closely related infrastructure role. Strong proficiency in Python for ML pipeline development, tooling, and automation. Hands-on experience with ML pipeline orchestration tools, particularly Apache Airflow. Experience with model registries and experiment tracking platforms (MLflow preferred). Experience deploying and operating ML workloads on AWS (Batch ...

code for data workloads (Terraform or Bicep) * Databricks - Delta Lake, Unity Catalog, job cluster vs interactive cluster trade-offs, cost-aware compute management, Spark job optimisation * Relational databases: PostgreSQL at production scale - query optimisation, indexing strategies, table partitioning, replication, schema design for both OLTP and analytical workloads * MongoDB - document … documented data products * Nice to Have * Financial markets data - market data feeds (Bloomberg, Refinitiv), tick data, trade history, reference data, or instrument master management * Apache Spark or Flink for large-scale stream and batch processing beyond the Databricks ecosystem * dbt or equivalent SQL transformation layer; experience building ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

keep our business moving, bringing fresh thinking and technical flair to every challenge. Your voice will shape our data design principles, your ideas will spark conversation across engineering forums, and your expertise in analytics and operational processing will help unlock smarter, faster decision-making. Key Actions Work with Data … years’ experience across BI, data warehousing, data management, ETL and database technologies Strong hands-on experience building and optimising data pipelines using Python, SQL, Apache Spark, and Microsoft SSIS Proven ability to design, deliver and maintain scalable data solutions across data warehouses and Big Data Lakes, supporting both ...

experience building and deploying machine learning and AI solutions in production. Hands‐on experience with: Python (PyTorch, TensorFlow, scikit‐learn) Data engineering tools (SQL, Spark, Airflow or similar) LLMs, NLP, and document intelligence (e.g. RAG, embeddings) Experience with cloud platforms (AWS, Azure, or GCP). Understanding of MLOps ...

Skills & Experience* Proven experience delivering enterprise-scale Databricks solutions end-to-end.* Strong background in Lakehouse Architecture, including structured and unstructured data.* Expertise in Spark, PySpark, Delta Lake, and Databricks workflows.* Experience building scalable ETL/ELT pipelines, including Delta Live Tables.* Strong programming skills in Python, Scala ...

Vault . Hands-on experience exposing Lakehouse solutions to BI tools (e.g. Power BI , Tableau ). Proficiency in Databricks Workflows , Delta Live Tables , and Spark for batch and streaming pipelines. Strong programming capability in Python , Scala , and/or SQL . Practical experience with at least one major cloud ...

commodities markets or broader financial markets. Knowledge of quantitative modeling, risk management, or algorithmic trading. Familiarity with big data technologies like Kafka, Hadoop, Spark, or similar. Why Work With Us? Impactful Work: Directly influence the profitability of the business by building technology that drives trading decisions. Innovative Culture ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Improve engineering standards, processes, and tooling ...

auditable, lineage-aware, idempotent, and resilient in regulated environments. Implement robust data quality controls including testing, validation, monitoring, and alerting. Drive performance optimization across Spark and Snowflake workloads. Partner with Product, Engineering, DevOps, and Regulatory teams to translate requirements into scalable technical designs. Contribute to the continuous improvement ...

warehousing concepts Familiarity with version control systems, particularly Git Desirable Skills: Experience with infrastructure as code tools such as Terraform or CloudFormation Exposure to Apache Spark for distributed data processing Familiarity with workflow orchestration tools such as Airflow or AWS Step Functions Understanding of containerisation using Docker Experience ...

accommodation during the recruitment process, please let us know. Why This Role Matters Ignite, Innovate, Impact. As an AI Engineer in the AI Spark team - a focused, high‐impact product innovation team responsible for accelerating the adoption of AI‐driven capabilities across Hansen products, you will play a pivotal ...

Azure or AWS Strong experience designing and delivering data solutions in Databricks Proficient with SQL and Python Experience using Big Data technologies such as Apache Spark or PySpark Great communication skills, effectively participating with Senior Stakeholders Nice to haves: Azure/AWS Data Engineering certifications Databricks certifications What ...

followed Required Skills & Experience: Strong experience with Azure data platform (e.g., Data Factory, Data Lake, Synapse) Hands-on expertise with Databricks (PySpark or Spark) Solid SQL development skills Experience working with Kafka for streaming data pipelines Good understanding of lakehouse architecture, including bronze/silver/gold layering concepts ...