1 to 25 of 49 Apache Spark Jobs in London

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

modern AI systems. Strong programming experience in Python. Data & Infrastructure Experience with some of the following technologies is highly desirable: Frameworks/Infrastructure Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Libraries Pandas Polars Engineering ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements … Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

with structured streaming, Delta Lake and Delta Live TablesProven experience in Lakehouse migration and modernization using open table formats such as Delta Lake or Apache IcebergProficiency with cloud-native services on Azure and knowledge of multi-cloud environments (AWS or GCP)Hands-on experience with CI/… governance, security practices and enterprise integration controlsBackground in financial trading data or related domainsOfficial Databricks certifications such as Certified Data Engineer Professional or Apache Spark DeveloperWe offerEPAM Employee Stock Purchase Plan (ESPP)Protection benefits including life assurance, income protection and critical illness cover Private medical insurance and dental ...

machine learning or data science.Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials.Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives).Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow).Strong experience ...

Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version control, CI/CD, testing ...

Knowledge of data modelling, data warehousing and Lakehouse architectures . Experience with data quality, governance and CI/CD pipelines . Exposure to Kafka, Spark Streaming or real-time data processing would be advantageous. Requirements: Ideally 7+ years of commercial Data Engineering experience. Strong hands-on technical background ...

designing data platforms using AWS, Azure or Google Cloud.Expertise in data modelling, SQL, NoSQL and modern database technologies.Experience with big data technologies, such as Apache Spark, Hadoop or distributed computing frameworks.Programming knowledge in Python, Scala, Java or similar languages.Experience architecting AI and machine learning solutions, including deploying models ...

requiredDo not rule yourself out if you do not have these. They are a bonus, not a checklist. Experience with Delta Lake, Iceberg, Spark or Databricks, PalantirExperience optimising LLM serving with tools such as vLLM, TGI or TensorRT LLMSearch and ranking experience. For example Elasticsearch or rerankersBackground in time ...

migration strategies as well as with machine learning, LLMs, and agentic AI workflows. Proficient in programming, debugging, and problem-solving using SQL, Python, Apache Spark, cloud platforms, and Lakehouse architectures and with AI tools. Experience collaborating with Global System Integrators (GSIs) and third-party consulting organizations to drive ...

Python and SQL with strong knowledge of data modelling and ETL/ELT processes Solid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processing Experience with Azure Data Factory (ADF), Qlik Replicate and modern data architecture practices Excellent communication, stakeholder management ...

production rather than for novelty. You don't need to know all of it on day one. Data and compute: S3, Athena, Spark on EMR, SQLOrchestration and MLOps: Airflow, MLflow, Docker, ECS, Terraform, GitHub Actions for CI/CDModels: foundation models served through Amazon BedrockAgents: LangGraph for orchestration, deployed ...

experience building, deploying, and scaling AI/ML solutions using frameworks and platforms such as Scikit-learn, PyTorch, TensorFlow, Hugging Face, LangChain, Databricks, Spark, vector databases, cloud services (AWS/Azure), and MLOps tools including MLflow and Kubeflow. Proven ability to develop production-grade, large-scale AI systems with ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake).Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain).Hands-on experience working with Distributed Spark based systemsExperience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external clients across … variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCommunication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core Traits: Passion for lifelong ...

data governance, ethics and responsible AI Eligibility for Enhanced DBS and BPSS clearance Desirable: Experience with PyTorch, TensorFlow, LangGraph, R or Java Experience with Spark, Databricks or similar distributed data platforms, and exposure to MLOps Experience of AWS, Azure or GCP AI/ML services, ideally with vendor certifications ...

time ML & reproducibility: A strong grasp of production inference patterns and reproducible environments (Docker, MLflow or equivalent). Cloud & tooling: AWS, plus SageMaker, Snowflake, Spark/Databricks and Kubernetes. Engineering mindset: A focus on reliability, maintainability and continuous improvement. ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...