1 to 25 of 39 Apache Spark Jobs in the South East

help design and deliver cutting-edge data solutions for customers operating in some of the UK's most complex and challenging environments. Using Databricks, Apache Spark and cloud-native technologies, you’ll develop scalable Lakehouse architectures and production-grade data pipelines that enable organisations to trust, share … opportunity to grow your career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable ...

Responsibilities: Design, build and enhance a secure, scalable and reliable Databricks platform Develop and maintain batch and real-time data pipelines using Databricks, Apache Spark and Delta Lake Contribute to platform architecture, engineering standards and best practices across the data estate Work with Product, Analytics and Data Science … continuous improvement across the engineering team Job Requirements: Significant experience building and supporting scalable, production-grade data platforms Strong hands-on expertise with Databricks, Apache Spark, Delta Lake, Python and SQL Good understanding of cloud-based data engineering, ideally within Microsoft Azure environments Experience with Infrastructure as Code ...

Databricks Senior Data Engineer - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Senior Data Engineer (Databricks) - London (Hybrid) - Up to £85,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a fast-growing technology business that's investing heavily in its data platform and engineering capability. This is an opportunity to join a highly collaborative team … progression within a growing technology business. Key Experience Required Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python, and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

Testing Guild with common standards such as metrics and use of test tools Here’s a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT We’d love to hear from ...

common technical standards and ways of working, and presenting these to the community Here’s a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT What we’re looking for We’d love ...

risk datasets. Key Responsibilities Design, develop and maintain scalable data pipelines using Python, PySpark and Databricks Build and optimise ETL/ELT workflows using Spark and Delta Lake Engineer and transform data supporting Counterparty Credit Risk, Credit Risk and wider risk functions Work with datasets relating to areas such … models and architectures across Lakehouse/Medallion environments Develop and manage Databricks notebooks, jobs and workflows Optimise large-scale distributed data processing and troubleshoot Spark performance Implement strong data quality, reconciliation, governance and monitoring controls Build reliable logging and alerting across production data pipelines Integrate data platforms with Power ...

with strong Foundry depth). Proven, hands-on experience working with Palantir Foundry in a production environment. Strong proficiency in Python, SQL, PySpark and Spark SQL Experience delivering production pipelines in AWS, Azure, or GCP environments. Solid understanding of data modeling, schema design, metadata management, and governance. Familiarity with ...

fields of data analytics and software Proficiency in Python programming language and common data science packages. Experience with data engineering tools (e.g. Apache Spark, AWS Glue, Kinesis, Snowflake, Apache Airflow) Experience in building data pipelines and data streaming applications. Experience with tools to support the software ...

more data-driven culture. Description Design, develop, and maintain scalable data pipelines using Python and Databricks. Build and optimise ETL/ELT workflows using Spark and Delta Lake. Develop and maintain data models and lakehouse architectures. Ensure high levels of data quality, reliability, governance, and security. Optimise the performance … Data Engineering. Strong commercial experience with Python. Advanced hands-on knowledge of Databricks, including notebooks, workflows, jobs, and Delta Lake. Strong experience with Apache Spark/PySpark. Proven track record building and maintaining scalable ETL pipelines. Strong SQL and data modelling expertise. Experience working within AWS, Azure ...

expertise: Platform Experience: Strong, hands-on experience with Databricks or Snowflake on AWS. Programming & Orchestration: Proven Python skills, including deep expertise in PySpark (or Spark) and Airflow. Data Modelling: Advanced SQL skills and extensive experience with end-to-end data modelling. Data Integration: Proven experience building both batch ...

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

unstructured data from multiple sources, including telemetry, operational datasets, OSINT and SOCMINT. Key Responsibilities Design, build and maintain scalable data pipelines using Azure Databricks, Spark/PySpark and Azure Data Lake Storage Ingest, transform and process large volumes of structured, semi-structured and unstructured data from multiple sources Develop … Excellent Python/PySpark and SQL data engineering skills Strong experience designing and developing ETL/ELT and large-scale data pipelines Experience with Apache Spark and Databricks Lakehouse architectures Strong knowledge of Azure Data Lake Storage (ADLS) Experience integrating Databricks with wider Azure data and analytics services ...

similar data-focused role Strong SQL and Python development skills Experience building and supporting scalable data pipelines and transformation processes Hands-on experience with Apache Spark, Databricks, dbt and cloud-based data platforms Good understanding of data modelling principles, data lakes and data warehouse architectures Experience working with … Essential: Degree in Computer Science, Software Engineering, Data Engineering, Information Systems, Mathematics or a related discipline; or equivalent practical experience Desirable: Relevant certifications in Spark, cloud technologies, data engineering or software engineering practices Why should you apply? We're not a perfect place - but we're a people place. ...

critical data ecosystem while supporting the delivery of high-value data products. Key Responsibilities Design, build and maintain scalable data pipelines using Databricks and Apache Spark. Develop and optimise batch and streaming data workflows, ensuring reliability, scalability and performance. Contribute to the design and implementation of modern Lakehouse data ...

experience working with Data A strong understanding of data integration techniques and solution/performance optimisation Excellent understanding and knowledge of SQL, Python or Spark Apply relevant data modelling techniques (e.g. STAR, Snowflake) Excellent Communication and Teamwork Skills Coaching and mentoring experience Technologies & Experience Desired: Git and JIRA software ...

Designed and delivered modern data platforms end to end on AWS and/or Databricks Still writes production-grade code: strong Python and SQL, Spark, dbt or equivalent, infrastructure-as-code (Terraform preferred) Track record in a consultancy, systems integrator or other client-facing setting comfortable being accountable ...

engineering practices . Responsibilities will include: Providing Staff-level technical leadership across complex data engineering initiatives Remaining deeply hands-on with Python, PySpark and Spark Modernising and migrating large-scale data pipelines across AWS Designing and challenging approaches across AWS Glue, Airflow and associated technologies Shaping scalable data architectures … quality and delivery speed Raising engineering standards and capability across multiple teams What we're looking for Deep hands-on Python/PySpark and Apache Spark expertise Strong AWS data engineering , particularly AWS Glue Strong Apache Airflow experience Experience delivering complex data transformation or modernisation programmes Strong ...

stakeholder engagement, and a passion for national security. Ideal candidates will have knowledge in some of the following: ETL processing languages such as Python Apache Spark/NiFi/Kafka Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) AWS and data-related services Palantir Foundry ...

GenAI infrastructure and workflows Skills and experience: Passionfor building scalable, resilient cloud data platforms Strongexperience with Databricks or Snowflake on AWS ProvenPython skills, including Spark and Airflow expertise AdvancedSQL skills and end‐to‐end data modelling experience Experiencebuilding batch and real-time data integrations Hands-onCICD skills with ...

hands‐on credibility to review a design and challenge an approach. Practical experience with a public cloud lakehouse or modern data platform; Databricks and Spark experience is an advantage. Solid grounding in relational databases at scale, ideally including Oracle and/or PostgreSQL. Real experience of DevOps practice applied ...

join a growing data platform team. You'll play a key role in designing, building and optimising production-grade data pipelines across Databricks, Spark and Delta Lake, while helping shape the engineering standards and architecture behind the wider data platform. What you'll be doing: Build and optimise scalable … Databricks/Spark data pipelines across batch and streaming workloads. Develop Lakehouse solutions using Delta Lake and medallion architecture. Implement data quality, monitoring, governance and security across the platform. Work closely with Data Scientists, Analysts and Product teams to deliver high-quality data products. Contribute to engineering standards, code ...

Monitor and maintain - Monitor data systems for performance issues and make any necessary updates. What You’ll Bring ETL processing languages such as Python Apache Spark/NiFi/Kafka Relational databases (e.g., PostgreSQL, MySQL) and NoSQL databases (e.g., MongoDB, Cassandra) AWS and data-related services Palantir Foundry ...

problem-solver who takes initiative in complex production settings Experience with scientific computing, deep learning, big data, or health IT ontologies (e.g., PyTorch, JAX, Spark, HL7, FHIR) (desirable) Familiarity with cloud infrastructure (Azure/AWS), infrastructure as code, Kubernetes, Linux, Docker, data pipelines, and MLOps tools (desirable) Passion ...

databases, e.g. SQL Server, PostgreSQL, Oracle, MySQL; Data visualisation software: Spotfire, Tableau, or Power BI; Azure cloud computing platform; Big data technologies such as Spark, Elasticsearch, Hadoop; Statistical techniques (regression, clustering etc.); Machine learning and pattern recognition; Front-end web development e.g. HTML, JavaScript ...