1 to 25 of 257 Apache Spark Jobs in the UK

complex legacy data pipelines into production-grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high-quality testing frameworks covering performance, data quality and reliability Leveraging Agentic AI development tools to improve … consumed We're looking for someone with experience in Staff, Principal or Lead Data Engineering roles Python and expert-level PySpark AWS Glue Apache Spark Apache Airflow Cloud-native Data Engineering architectures Large-scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing, validation ...

media metadata platform - Media Content Catalogue. You will be involved with the design, scaling, and optimisation of this modern, cloud-scale lakehouse built on Apache Spark, AWS Glue, and Apache Iceberg/Parquet, transforming complex broadcast & archival media metadata into structured, aggregated, unified, high-performance data assets. … Demonstrable experience of delivering data-powered applications in production - Hands-on experience with modern cloud data analysis & engineering (we use SQL, Python, AWS Glue, Apache Iceberg, Apache Parquet, Apache Spark) - Experience of modern data transformation pipeline tools such as dbt or SQLMesh - Working experience of data ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure ...

platform at scale, ideally in a consumer or marketplace environment. Deep understanding of distributed systems and modern data ecosystems — including extensive experience with Databricks, Apache Spark, Apache Kafka, Apache Flink, Apache Beam and DBT. Demonstrated success in designing, building, and operating data platforms at scale ...

complex legacy data pipelines into production‐grade PySpark Building scalable, reusable data movement frameworks across multiple engineering teams Driving best practice around AWS Glue, Apache Spark and Apache Airflow Designing high‐quality testing frameworks covering performance, data quality and reliability Leveraging agentic AI development tools to improve … data is governed, managed and consumed Qualifications Staff, Principal or Lead Data Engineering roles Expertise in Python and PySpark Experience with AWS Glue, Apache Spark and Apache Airflow Cloud‐native data engineering architectures Large‐scale data pipeline modernisation Building reusable engineering patterns within enterprise environments Data testing ...

you. You'll be joining a large-scale cloud data transformation programme, helping design, build and optimise enterprise data platforms using Python, Databricks and Spark within a modern Lakehouse environment. Key Responsibilities Design, develop and maintain scalable data pipelines using Python and Databricks. Build, optimise and support enterprise …/ELT workflows using Apache Spark and Delta Lake. Design and develop robust data models and Lakehouse architectures. Implement and manage data workflows within the Databricks ecosystem. Ensure high levels of data quality, governance and pipeline reliability. Optimise performance across large-scale distributed data processing platforms. Collaborate with ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

root‐cause analysis. Experience building or supporting production data pipelines in a collaborative engineering environment. Experience working with distributed data processing frameworks such as Apache Spark . Working knowledge of common data formats such as JSON , Avro and Parquet . Technology Environment The role will involve working with … should bring relevant experience and the ability to work across comparable technologies. Examples of technologies in scope include: Data processing and logic: ANSI SQL, Apache Spark, Kafka Platforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQ Engineering and deployment: CI/CD tooling, containerised ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.What … lake & Fabric. Understand which Azure data focused components to use for which purpose.Microsoft certified architectDatabricksDesign and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.Ability to Design, Build and deploy AI/ ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. What … which Azure data focused components to use for which purpose. Microsoft certified architect Databricks Design and build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Ability to Design, Build and deploy ...

product managers, to understand data requirements and deliver high-quality solutions Architect and implement data ingestion, transformation, and storage processes using tools such as Apache Spark, Azure Data Factory, and other similar technologies Optimize data pipeline performance, ensuring data accuracy, reliability, and timely delivery Mentor and guide junior … Engineering, or a related field 7+ years of experience in data engineering, with at least 3 years in a leadership role Strong proficiency in Apache Spark, Azure Data Factory, and other data pipeline tools Experience with cloud platforms such as Azure, AWS, or GCP Solid understanding of data ...

scale data processing. Proficiency in Python/SQL for data engineering and scripting. Familiarity with risk analytics environments or financial data. Strong experience with Apache Spark (Spark Engine), including performance optimization and distributed data processing. Strong experience ingesting data from diverse sources including relational databases, REST APIs ...

technologies, share knowledge, and contribute to a culture of excellence and innovation. Key responsibilities include: • Design & Build: Develop and optimise data pipelines using Databricks, Apache Spark, and Python. • Develop & Deliver: Create scalable data solutions on AWS leveraging S3, Glue, Lambda, and related services. • Integrate & Automate: Implement ETL processes … Hands-on commercial expertise with Databricks. You should have: • Proven experience as a Data Engineer working with large, complex datasets. • Hands-on expertise with Apache Spark, and SQL. • Strong proficiency in Python (PySpark preferred). • Experience with AWS cloud services including S3, Glue, Lambda, IAM. • Familiarity with ...

designing and delivering modern data platforms, working closely with clients and technical teams to build scalable, secure and high-performing solutions using Databricks, Apache Spark and cloud technologies. Key Responsibilities Design and architect Databricks-based data platforms and solutions. Build scalable data lakehouse and data warehouse architectures. Design … Databricks and modern data platform delivery. Skills & Experience Required 7+ years' experience in Data Architecture or Solution Architecture. Strong commercial experience with Databricks and Apache Spark . Experience delivering enterprise data platforms on AWS, Azure or GCP. Strong understanding of data warehousing, data lakes and lakehouse architectures. Experience ...

frameworks, tools and applications that make that data available to other teams and systems. Your primary toolset in your work will be Java microservices, Spark/Scala data processing (also some Flink), Kubernetes and AWS native tooling.**What You'll Do:*** Write highly fault-tolerant Java code within Apache Spark to produce platform products used by our customers to query our event pipelines/ingestion for insight into active threat trends and related analytics* Design, develop, and maintain ultra-high-scale data platforms that process petabytes of data* Participate in technical reviews of our products and help ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain). Hands-on experience working with Distributed Spark based systems Experience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external … clients across a variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred [Preferred] Experience working with Apache Spark to process large-scale distributed datasets Communication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core ...

production-grade solutions using BigQuery, Dataflow, Dataproc, Pub/Sub, and Cloud Composer . Frameworks & Storage: Strong understanding of batch/streaming tools (e.g., Apache Spark, Apache Beam ) and when to apply different storage paradigms (relational, columnar, document, data lakes). Modern Best Practices: Active practice with ...

operating data pipelines and analytical systems using technologies such as Google BigQuery, Amazon Athena, or ClickHouse. Experience with distributed data processing frameworks such as Apache Spark and/or Apache Flink. Experience with messaging and streaming systems such as Apache Kafka or Apache Pulsar. Experience ...

operating data pipelines and analytical systems using technologies such as Google BigQuery, Amazon Athena, or ClickHouse. Experience with distributed data processing frameworks such as Apache Spark and/or Apache Flink. Experience with messaging and streaming systems such as Apache Kafka or Apache Pulsar. Experience ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

Experience using Git/version control Strong communication skills with both technical and non-technical stakeholders Desirable Skills: Microsoft Fabric Azure Synapse Analytics Databricks Apache Spark/PySpark dbt Apache Airflow Power BI Azure DevOps CI/CD pipelines Terraform What's On Offer: £65k+ salary package ...

delivering enterprise-scale data platform solutions. A strong background in Solution Architecture or Technical Architecture. Hands-on experience with open-source technologies such as Apache Spark, Trino, Airflow, Apache Superset, or similar data engineering platforms. Experience working with cloud platforms including Azure, AWS, or GCP. Experience producing ...

learning or data science. Cloud certifications (e.g. Azure, GCP, AWS) and relevant AI/ML credentials. Experience with big data and distributed computing (e.g. Spark, Databricks, Hadoop or cloud-native alternatives). Experience with containerisation and DevOps/MLOps tools (e.g. Docker, Kubernetes, Git, CI/CD, MLflow, Airflow ...

understanding of Quantexa components such as: Entity Resolution (ER), Scoring Framework, Contextual Network Generation, UI Configuration. Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch. Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills ...