1 to 25 of 60 Apache Spark Jobs in London

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

related disciplines within enterprise environments 4+ years of hands-on experience designing and implementing solutions on the Databricks Lakehouse Platform Strong expertise in Apache Spark, SQL, Python, Delta Lake, and modern data engineering practices Experience building and optimizing Extract, Transform, Load and Extract, Load, Transform data pipelines … solutions on the Databricks Lakehouse Platform. Design and implement scalable data and analytics solutions leveraging Databricks technologies Build, maintain, and optimize data pipelines using Apache Spark, SQL, Python, and Delta Lake Develop governed datasets and data products that support reporting, analytics, and artificial intelligence initiatives Partner with business ...

unknowns early and propose pragmatic solutions.Data Engineering• Design and build scalable, resilient and maintainable enterprise data solutions.• Apply strong engineering practices using Databricks, Apache Spark/PySpark, Python, SQL and modern lakehouse technologies.• Develop reusable patterns for data ingestion, transformation, integration and delivery.• Design for incremental processing, schema … experience developing production-quality Databricks notebooks using Python and SQL.• Strong knowledge of modern data engineering, distributed processing and Lakehouse concepts.• Experience with Apache Spark or comparable large-scale data technologies.• Experience designing and implementing cloud-based data platforms and products.• Strong knowledge of data modelling, ingestion, transformation ...

hands-on Senior Databricks Solutions Architect to support a complex, high-performance data platform within a global organisation. Youll design and optimise Spark workloads, build production streaming pipelines and deliver scalable Databricks solutions using Delta Live Tables, Workflows, Unity Catalog, Python, PySpark, Terraform and Databricks Asset Bundles. Key requirements … Advanced Apache Spark and Spark SQL optimisation Production Spark Structured Streaming experience Deep Databricks platform knowledge Strong Python and PySpark engineering capability Delta Live Tables, Workflows and Unity Catalog Terraform and Databricks Asset Bundles CI/CD and GitHub Actions Ability to work autonomously ...

Skills & Experience Data Engineering & Cloud Extensive experience designing, building and operating production data platforms using Azure Databricks and Azure. Strong hands-on expertise with Apache Spark, PySpark, Spark SQL and Delta Lake. Experience implementing and managing Unity Catalog, including governance, lineage and access controls. Proven delivery ...

Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands ...

within either Flask, Tornado or Django, DockerExperience working with ETL pipelines is desirable e.g. Luigi, Airflow or ArgoExperience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data qualityPreparing data for predictive and prescriptive modellingHands on coding ...

modern frameworksWorking knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least oneDeep experience with distributed computing with Apache Spark and knowledge of Spark runtime internalsFamiliarity with CI/CD for production deploymentsWorking knowledge of MLOps, ML/AI models ...

while collaborating with the Data & Analytics Lead, technical colleagues and business stakeholders. Experience in the following areas would also be useful: Materialised Lake Views Apache Spark or Python. Power BI semantic modelling. APIs and cloud business system integration. Data governance, lineage and quality frameworks. Professional services experience. What ...

Excellent Python and SQL Solid grasp of data modelling, warehousing and lakehouse concepts Hands-on with orchestration (e.g. Airflow, Dagster) and distributed processing (e.g. Spark) Experience with cloud data platforms (AWS, Azure or GCP) and containerised deployment (Docker, Kubernetes) Good engineering practice: version control, CI/CD, testing ...

Senior Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD skills ...

Knowledge of data modelling, data warehousing and Lakehouse architectures . Experience with data quality, governance and CI/CD pipelines . Exposure to Kafka, Spark Streaming or real-time data processing would be advantageous. Requirements: Ideally 7+ years of commercial Data Engineering experience. Strong hands-on technical background ...

Skills & Experience Strong commercial experience in data engineering within cloud-based environments. Advanced Python and SQL development skills. Hands-on experience with Azure Databricks, Apache Spark and PySpark. Good understanding of Delta Lake, lakehouse architecture and modern data modelling. Experience building and managing production data pipelines, including batch ...

requiredDo not rule yourself out if you do not have these. They are a bonus, not a checklist. Experience with Delta Lake, Iceberg, Spark or Databricks, PalantirExperience optimising LLM serving with tools such as vLLM, TGI or TensorRT LLMSearch and ranking experience. For example Elasticsearch or rerankersBackground in time ...

migration strategies as well as with machine learning, LLMs, and agentic AI workflows. Proficient in programming, debugging, and problem-solving using SQL, Python, Apache Spark, cloud platforms, and Lakehouse architectures and with AI tools. Experience collaborating with Global System Integrators (GSIs) and third-party consulting organizations to drive ...

Ingestion projects in Azure environment. Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment. Experience on Python scripting, Spark SQL PySpark is a plus. We use generative AI tools to support our candidate screening process. This helps us ensure a fair, consistent ...

Python and SQL with strong knowledge of data modelling and ETL/ELT processes Solid understanding of Delta Lake, Delta Live Tables, Unity Catalog, Spark Structured Streaming and large-scale data processing Experience with Azure Data Factory (ADF), Qlik Replicate and modern data architecture practices Excellent communication, stakeholder management ...

production rather than for novelty. You don't need to know all of it on day one. Data and compute: S3, Athena, Spark on EMR, SQLOrchestration and MLOps: Airflow, MLflow, Docker, ECS, Terraform, GitHub Actions for CI/CDModels: foundation models served through Amazon BedrockAgents: LangGraph for orchestration, deployed ...

experience building, deploying, and scaling AI/ML solutions using frameworks and platforms such as Scikit-learn, PyTorch, TensorFlow, Hugging Face, LangChain, Databricks, Spark, vector databases, cloud services (AWS/Azure), and MLOps tools including MLflow and Kubeflow. Proven ability to develop production-grade, large-scale AI systems with ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake).Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain).Hands-on experience working with Distributed Spark based systemsExperience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external clients across … variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCommunication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core Traits: Passion for lifelong ...

Maintain the controls and records needed to support auditing, data traceability and compliance requirements. Performance and operational management Investigate and improve the performance of Spark workloads, Delta tables and SQL compute. Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure. Set up monitoring and alerting … knowledge sharing and mentoring across the team. What we are looking for Practical experience managing Databricks in a production environment. A strong understanding of Apache Spark, Delta Lake and Unity Catalog. Experience working with one or more major cloud providers, including storage, networking and identity management. Confidence using ...

data governance, ethics and responsible AI Eligibility for Enhanced DBS and BPSS clearance Desirable: Experience with PyTorch, TensorFlow, LangGraph, R or Java Experience with Spark, Databricks or similar distributed data platforms, and exposure to MLOps Experience of AWS, Azure or GCP AI/ML services, ideally with vendor certifications ...

time ML & reproducibility: A strong grasp of production inference patterns and reproducible environments (Docker, MLflow or equivalent). Cloud & tooling: AWS, plus SageMaker, Snowflake, Spark/Databricks and Kubernetes. Engineering mindset: A focus on reliability, maintainability and continuous improvement. ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...