1 to 25 of 31 Apache Spark Jobs in the South East

Data Engineer 6 Month Contract Location: London (Onsite 4 days a week) Rate: £700 - £800 per day (INSIDE IR35) Skills: Python, Azure, Kubernetes, Apache Airflow, Apache Spark, DV Clearance We are looking to recruit a DV Cleared Data Engineer for a leading technology consultancy. This will … Work across Azure environments on the low side Support data engineering workloads within Kubernetes environments on the high side Develop and maintain workflows using Apache Airflow Work with Apache Spark for large-scale data processing Support the ongoing development and improvement of established data pipelines Help ensure ...

help design and deliver cutting-edge data solutions for customers operating in some of the UK's most complex and challenging environments. Using Databricks, Apache Spark and cloud-native technologies, you’ll develop scalable Lakehouse architectures and production-grade data pipelines that enable organisations to trust, share … opportunity to grow your career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable ...

Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platform Develop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta Lake Contribute to platform architecture, engineering standards and best practices across the data estate Work with Product, Analytics and Data Science … continuous improvement across the engineering team Job Requirements: Significant experience building and supporting scalable, production-grade data platforms Strong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQL Good understanding of cloud-based data engineering, ideally within Microsoft Azure environments Experience with Infrastructure as Code ...

Databricks Senior Data Engineer - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

common technical standards and ways of working, and presenting these to the community Here’s a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT What we’re looking for We’d love ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

more data-driven culture. Description Design, develop, and maintain scalable data pipelines using Python and Databricks. Build and optimise ETL/ELT workflows using Spark and Delta Lake. Develop and maintain data models and lakehouse architectures. Ensure high levels of data quality, reliability, governance, and security. Optimise the performance … Data Engineering. Strong commercial experience with Python. Advanced hands-on knowledge of Databricks, including notebooks, workflows, jobs, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and maintaining scalable ETL pipelines. Strong SQL and data modelling expertise. Experience working within AWS, Azure ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

similar data-focused role Strong SQL and Python development skills Experience building and supporting scalable data pipelines and transformation processes Hands-on experience with Apache Spark, Databricks, dbt and cloud-based data platforms Good understanding of data modelling principles, data lakes and data warehouse architectures Experience working with … Essential: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems, Mathematics or a related discipline; or equivalent practical experience Desirable: Relevant certifications in Spark, cloud technologies, data engineering or software engineering practices Why should you apply? We're not a perfect place - but we're a people place. ...

critical data ecosystem while supporting the delivery of high-value data products. Key Responsibilities Design, build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimise batch and streaming data workflows, ensuring reliability, scalability and performance. Contribute to the design and implementation of modern Lakehouse data ...

hands‐on credibility to review a design and challenge an approach. Practical experience with a public cloud lakehouse or modern data platform; Databricks and Spark experience is an advantage. Solid grounding in relational databases at scale, ideally including Oracle and/or PostgreSQL. Real experience of DevOps practice applied ...

evolving platform stack — from data ingestion and event processing to REST services and internal tooling. Build and maintain streaming and batch processing systems (Flink, Spark, Kafka) that power real‐time and analytical workloads. Take responsibility for all aspects of software engineering: design, implementation, QA, and maintenance, with … Willingness and ability to take on new technologies as our stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Familiarity with additional technologies: GraphQL, React, HTML5, JavaScript, CSS, Postgres ...

join a growing data platform team. You'll play a key role in designing, building and optimising production-grade data pipelines across Databricks, Spark and Delta Lake, while helping shape the engineering standards and architecture behind the wider data platform. What you'll be doing: Build and optimise scalable … Databricks/Spark data pipelines across batch and streaming workloads. Develop Lakehouse solutions using Delta Lake and medallion architecture. Implement data quality, monitoring, governance and security across the platform. Work closely with Data Scientists, Analysts and Product teams to deliver high-quality data products. Contribute to engineering standards, code ...

Monitor and maintain - Monitor data systems for performance issues and make any necessary updates. What You’ll Bring ETL processing languages such as Python Apache Spark/NiFi/Kafka Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) AWS and data-related services Palantir Foundry ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

forecasting and predictive modelling. Advanced Python skills and experience with common ML libraries. Strong SQL skills. Experience working with Databricks and/or Spark environments. Ability to translate business requirements into practical AI and ML solutions. Experience communicating technical concepts to business stakeholders. Strong problem-solving skills with ...

scale data and analytics platforms Expertise in cloud architectures (preferably Azure, incl. Azure Data Factory, Synapse, Databricks, Fabric, etc.) Experience with big data technologies (Spark, Hadoop, distributed systems) Proficiency in data modelling, especially time-series and simulation datasets Knowledge of APIs, microservices, and event-driven architectures Energy Domain Knowledge ...

model validation. Experiment tracking and reproducibility tools, for example MLflow, Weights & Biases or equivalent. Data wrangling and analysis using Pandas, NumPy, SQL; familiarity with Spark or similar is a plus. Model optimisation and research-to-prototype deployment fundamentals, including ONNX, TorchScript, FastAPI/gRPC and GPU acceleration, with CUDA ...

Limited (DDIL) communications environments, supporting local processing at the edge Building streaming ingestion, event pipelines, and batch/stream processing workloads using technologies like Apache Flink and/or Apache Spark Implementing metadata, lineage, and catalogue capabilities along with policy-based security and governance Job Requirements: Extensive ...

Ensure data reliability, scalability, and quality. Integrate data sources into a unified platform. Monitor and troubleshoot data workflows. Skills Required: Expertise in Azure Databricks, Spark, and big data tools. Strong SQL skills Knowledge of Python, ETL processes and cloud technologies. Familiarity with CI/CD practices for data pipelines. ...

/B tests and experimentation frameworks . Familiarity with structured and unstructured datasets. Excellent stakeholder management and communication skills. Desirable Skills Deep Learning NLP Spark AWS, Azure, or GCP Feature Stores Real-time Personalisation Systems Search and Ranking Technologies If you have a strong background in Machine Learning, recommendation ...