51 to 75 of 297 Apache Spark Jobs in England

with the Lead Data Engineer and partnering with data science, analytics and product teams. Key responsibilities * Build and maintain scalable data pipelines using Databricks, Apache Spark and Delta Lake * Develop and optimise batch and streaming workflows for reliability and performance * Contribute to lakehouse architecture and medallion pattern implementation … optimisation opportunities * Translate business requirements into technical solutions * Contribute to documentation and continuous improvement Required experience * Strong experience delivering production grade Databricks pipelines * Strong Spark, Delta Lake and Databricks Workflows experience * Python and SQL proficiency * Experience on Azure data platforms such as Azure Data Factory, ADLS Gen2, Azure DevOps ...

engineering teams. Strong Python and SQL skills, with experience working with large-scale datasets and distributed data processing environments. Experience with technologies such as Apache Spark and modern data engineering frameworks. Experience with cloud platforms and data services, ideally AWS, including technologies such as Redshift and S3. Experience ...

Azure, GCP).Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon).Familiarity with data lake architectures and distributed processing frameworks (e.g., Spark, Hadoop).Experience with version control tools (GitHub, Bitbucket) and CI/CD pipelines. Understanding of data governance, security, and compliance frameworks. Exposure to AI/ ...

Working knowledge of two or more common Cloud ecosystems (AWS, Azure, GCP) with expertise in at least one Deep experience with distributed computing with Apache Spark and knowledge of Spark runtime internals Familiarity with CI/CD for production deployments Working knowledge of MLOps, ML/ ...

/CD, Application Resiliency, and SecurityIn-depth knowledge of the financial services industry and their IT systemsPractical cloud native experiencePreferred qualifications, capabilities, and skillsAdvanced Apache Spark experience (PySpark/Spark SQL), including performance tuning (partitioning, shuffle optimization, caching), troubleshooting, and designing scalable batch/stream processing patterns. ...

Management Stakeholder Collaboration Leadership & Mentorship Technology Stack The successful candidate will have experience working with some or all of the following technologies: Data Processing: Apache Spark/PySpark Workflow Orchestration: Apache Airflow Containerisation: Kubernetes Cloud Platform: Microsoft Azure Data APIs & Semantic Layer: CubeJS Programming Languages: Python ...

large-scale ETL development. Proficiency in SQL and programming languages such as Python, Java, or Scala. Hands‐on experience with big data frameworks (e.g., Spark, Hadoop, Flink) and cloud platforms (AWS, GCP, or Azure). Familiarity with data warehouse technologies (e.g., Snowflake, Redshift, BigQuery) and workflow orchestration tools (e.g. ...

maintain SC clearance. AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...

messaging architectures. Data Platforms & Cloud Technologies Hands-on experience with: Azure Data Lake Databricks Snowflake Delta Lake Azure Synapse Analytics/Microsoft Fabric Spark/PySpark Strong understanding of: Data Lake and Lakehouse architecture Hybrid cloud environments Enterprise integration patterns Modern data formats including Parquet, Avro, and ORC DevOps ...

maintain SC clearance. Desirable AWS certification at Associate or Professional level Experience with streaming and event‐driven data technologies Experience with technologies such as Spark, Kafka or Databricks Experience delivering data platforms within public sector, defence or other regulated environments Experience working within secure environments and understanding associated security ...

. Knowledge of data modelling methodologies (star schemas, Data Vault, Kimball, Inmon). Familiarity with data lake architectures and distributed processing frameworks (e.g., Spark, Hadoop). Experience with version control tools (GitHub, Bitbucket) and CI/CD pipelines. Understanding of data governance, security, and compliance frameworks. Exposure ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

basis to support a MoD-owned data platform. The role focuses on building and maintaining secure data pipelines using Python, Azure and Kubernetes, with Apache Airflow and Apache Spark integrated into the stack. You will work within a government-aligned, secure environment, ensuring data remains hosted ...

Company Description Company DescriptionPublicis Media harnesses the power of modern media through leading agency brands Dysrupt, Infinite Roar, Publicis Collective, Publicis Health Media, Spark Foundry, Starcom and Zenith , as well as global accelerator PMX ; immersive experience group PMCI and access to integrated platform-based technologies and offerings from … marketing platforms. Build and optimize schemas, transformations, and ETL logic for high-performance, reusable data workflows. Contribute directly to the DFP codebase (Python, SQL, Spark) — developing pipelines and infrastructure alongside the engineering team. Establish data versioning, testing, and governance frameworks ensuring reliability, lineage, and compliance. Data Engineering Leadership Drive ...

Skills MUST HAVE: Strong hands-on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

Skills MUST HAVE: Strong hands‐on experience with Quantexa implementation and development. Proven experience building and supporting big data platforms . Advanced knowledge of Apache Spark/PySpark . Strong programming skills in Scala and/or Python . Expertise in SQL and large-scale data modelling. Strong ...

understand where traditional patterns remain appropriate – and where modern approaches add measurable value. Data Engineering & Platform Understanding: Strong knowledge of distributed processing (e.g. Apache Spark).Familiarity with Hadoop ecosystem components. Working knowledge of SQL and NoSQL technologies. Understanding of ETL/ELT design trade-offs. Awareness of DevOps ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

understanding of Terraform enterprise. Experience in delivering system design, application development, testing, and operational stability. Knowledge of Big Data distributed compute frameworks like Spark, Glue, MapReduce etc. Excellent troubleshooting, analytical, and communication skills. Preferred Qualifications, Capabilities, and Skills: Multi Region disaster recovery setup, monitoring and testing. Experience in Data ...

Risk, XVA, CCR, FRTB, portfolio risk, fixed income analytics, securitised products, liquidity risk, or managed risk-as-a-service solutions. Familiarity with Snowflake, Databricks, Spark, AWS, Azure, Confluence, Azure DevOps, Git, Tableau, Power BI, message queues, or comparable tools and platforms. Knowledge of market data vendors, data mastering, golden ...

Risk, XVA, CCR, FRTB, portfolio risk, fixed income analytics, securitised products, liquidity risk, or managed risk-as-a-service solutions. Familiarity with Snowflake, Databricks, Spark, AWS, Azure, Confluence, Azure DevOps, Git, Tableau, Power BI, message queues, or comparable tools and platforms. Knowledge of market data vendors, data mastering, golden ...

root-cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark. Working knowledge of common data formats such as JSON, Avro and Parquet. For More Experienced CandidatesStronger ownership of technical design across multiple datasets … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, KafkaData formats: JSON, Avro, ParquetPlatforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQEngineering and deployment: CI/ ...

Lead Data Engineer (Databricks) - London (Hybrid) - Up to £110,000 Databricks - Apache Spark - Delta Lake - Python - SQL - Azure We're partnering with a growing technology business investing heavily in its cloud data platform and AI capabilities. This is a senior, hands-on leadership role where … machine learning and large-scale data initiatives. Key Experience Required Proven experience as a Senior or Lead Data Engineer. Deep expertise in Databricks, Spark, Delta Lake, Python and SQL. Experience designing and scaling cloud-based data platforms. Strong knowledge of ETL/ELT, data modelling and distributed processing. Experience ...

decisions, documentation and continuous improvement initiatives. Key Skills & Experience Proven experience as a Data Engineer within production environments. Strong hands-on experience with Databricks, Apache Spark, Delta Lake, Python and SQL. Experience building and supporting scalable ETL/ELT pipelines. Strong understanding of distributed data processing and data ...

taking customers from EDW to the full capabilities of the Data Intelligence Platform. Demonstrated competence in the Lakehouse architecture including hands-on experience with Apache Spark, Python and SQL Excellent communication skills; both written and verbal Public Cloud Architecting: Hands-on experience designing and implementing big data architectures ...