51 to 75 of 106 Apache Spark Jobs in the UK

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

translate business needs into technical roadmaps What were looking for Strong experience in data architecture & large-scale systems design Expertise across Python, SQL, Spark, Scala Deep knowledge of AI/ML frameworks and modern data platforms Experience with cloud-native architecture and DevOps (Kubernetes, Terraform, CI/CD) Proven ...

Experience auditing and improving existing ML setups * Strong MLOps experience (model versioning, monitoring, CI/CD) * Experience working with data pipelines (ETL/ELT, Spark or similar) * Comfortable working with messy, real-world data and complex business logic * Experience in small, fast-moving teams (startup, scaleup, or consulting) * Strong ...

extracting, manipulating, and preparing data from relational databases. Experience with NoSQL databases is a plus. Big Data/Cloud: Experience with big data tools ( Spark, Hadoop ) and cloud computing platforms ( AWS, Azure, or GCP ) for scalable ML workflows. Visualization: Ability to create clear, compelling data visualizations using tools like ...

Docker and Kubernetes is a plus. Experience designing end to end ML data pipelines. Experience with database technologies such as PostgreSQL, MySQL, or MongoDB, Spark, Databricks and stream data processing such as Kafka, RedPanda is a plus. Direct experience in building and applying large-scale (100M+ users) machine learning ...

experience in software engineering, MLOps, and deploying machine learning models at scale.* Proficiency in SQL, Python, and familiarity with big data technologies (e.g., Kafka, Spark) and/or cloud platforms (e.g., GCP, AWS, or Azure).* Industry knowledge: Experience in gaming or digital entertainment is a strong plus. ...

stream data processing. Cloud Pub/Sub: For real-time messaging and event ingestion. Cloud Storage: For scalable and durable object storage. Cloud Composer (Apache Airflow): For workflow orchestration and pipeline management. Cloud Dataproc: For managed Hadoop and Spark clusters (good to have). Cloud SQL/Cloud ...

deliver intelligent data products, leveraging modern cloud and AI technologies. Key Responsibilities Design and implement robust data pipelines and ML workflows using Python, SQL, Spark, and Databricks. Develop and deploy machine learning models (including NLP, deep learning, and agentic AI) in production environments. Integrate data from diverse sources, including … Mentor junior engineers and contribute to internal knowledge sharing. Ensure solutions meet security, compliance, and performance standards. Qualification Core Data & AI Skills Python, SQL, Spark, Scala Machine Learning, NLP, Deep Learning, Prompt engineering, Agentic AI Data Architecture, Data Modelling, Data Engineering, Data Analysis DevOps & Engineering CI/ ...

will have strong experience in data engineering, with advanced proficiency in SQL and Python, alongside exposure to large-scale data processing frameworks such as Apache Spark. You should have a solid understanding of data architecture, data wrangling and data quality practices, as well as experience working within financial markets ...

Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring, logging ...

Lakehouse platform in Azure. Owning Databricks engineering, administration and platform optimisation. Designing end-to-end data platform solutions Leading implementation using Databricks, PySpark, Spark SQL and Azure Data Factory. Creating and maintaining CI/CD pipelines and automated testing frameworks. Developing APIs to support integration Implementing monitoring, logging ...

data engineering within distributed systems Proven ability to build and optimise data pipelines at scale Solid Python skills with exposure to tools such as Spark, PySpark, and Hadoop Strong understanding of data ingestion and batch processing methodologies Experience with text or unstructured data is advantageous but not essential ...

working Required Experience Extensive commercial experience with Java in production environments Strong experience designing and building Kafka‐based streaming platforms Hands‐on experience with Spark, Flink, Kafka Streams , or similar technologies Experience with Spring Boot and/or Python Deep experience working with AWS , including services such as Lambda ...

manage multiple priorities and respond calmly to issues. • Self-motivated with a collaborative approach. Preferred: • Familiarity with AWS data tooling such as S3, Athena, Spark or notebooks. • Experience supporting data products or client-facing datasets. • Understanding of release testing, UAT or production support processes. • Exposure to Jira/Agile ...

manage multiple priorities and respond calmly to issues. • Self-motivated with a collaborative approach. Preferred: • Familiarity with AWS data tooling such as S3, Athena, Spark or notebooks. • Experience supporting data products or client-facing datasets. • Understanding of release testing, UAT or production support processes. • Exposure to Jira/Agile ...

machine learning frameworks and programming languages such as Python, R, or similar. Experience working with large datasets and data processing technologies (e.g., Hadoop, Spark, SQL). Familiarity with anomaly detection, supervised, unsupervised learning methods, deep learning, and graph based solutions Demonstrated ability to work collaboratively in cross-functional teams ...

machine learning frameworks and programming languages such as Python, R, or similar. Experience working with large datasets and data processing technologies (e.g., Hadoop, Spark, SQL). Familiarity with anomaly detection, supervised, unsupervised learning methods, deep learning, and graph based solutions Demonstrated ability to work collaboratively in cross-functional teams ...

robust, scalable, and governed data solutions using modern cloud technologies. The Role Own the end-to-end design, build, optimisation, and support of scalable Spark/PySpark data pipelines (batch and streaming) Define and implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality … controls Design and manage secure data ingestion frameworks (e.g. Apache NiFi, APIs, SFTP/FTPS) for internal and external data sources Architect and maintain secure AWS-based data infrastructure (S3, IAM, KMS, Glue, Lake Formation, Lambda, Step Functions, CloudWatch, etc.) Implement orchestration using tools such as Airflow, Databricks Workflows ...

robust, scalable, and governed data solutions using modern cloud technologies. The Role Own the end-to-end design, build, optimisation, and support of scalable Spark/PySpark data pipelines (batch and streaming) Define and implement lakehouse architecture standards (medallion model: bronze, silver, gold), including governance, lineage, and data quality … controls Design and manage secure data ingestion frameworks (e.g. Apache NiFi, APIs, SFTP/FTPS) for internal and external data sources Architect and maintain secure AWS-based data infrastructure (S3, IAM, KMS, Glue, Lake Formation, Lambda, Step Functions, CloudWatch, etc.) Implement orchestration using tools such as Airflow, Databricks Workflows ...

influence within the team and communicate clearly with technical and non-technical stakeholders. Data Engineer (New Technology/Microsoft) Strong experience with Databricks (Spark, PySpark, Delta Lake, and Unity Catalog advantageous). Proficiency in Azure data services (Azure Data Factory, Data Lake, Azure Functions advantageous). Experience contributing … Engineer or Solutions Architect certification (desirable). Databricks Certified Data Engineer or Machine Learning Associate (desirable). Experience with streaming solutions (Kafka, Event Hubs, Spark Streaming) (desirable). Knowledge of machine learning and AI on Databricks (desirable). ...

expand their Devops team (multiple roles). Key requirements: Strong experience with AWS services and infrastructure as code using Terraform Data engineering expertise using Apache Spark and Python, including development in Jupyter Notebooks Workflow orchestration and containerisation experience with Apache Airflow and Docker Version control ...

Proven experience with modern data platforms such as Snowflake (AI Data Cloud) and cloud‐native services. Good understanding of open‐source table formats, specifically Apache Iceberg , to enable interoperability, schema evolution, and high‐performance analytics across multiple engines. Programming: Expert‐level proficiency in Java , Python and SQL . … Data & Orchestration: Hands‐on experience with Spark, Kafka, and orchestration tools like Apache Airflow, Dagster, or dbt. Data Modeling: Deep understanding of data warehousing and modern data lakehouse architecture. Leadership & Soft Skills Mentorship: Proven track record of upskilling junior and senior engineers. Communication: Ability to explain complex technical ...

data models and outputs Set up monitoring and ensure data health for outputs What we are looking for Proficiency in Python, with experience in Apache Spark and PySpark Previous experience with data analytics softwares Ability to scope new integrations and translate user requirements into technical specifications What ...

large-scale analytics platforms Expert knowledge of Quantexa, Entity Resolution (ER), Scoring Framework, Contextual Network Generation Expertise in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch Strong stakeholder management and communication skills - capable of interfacing with both business and technical audiences. The work is heavily weighted toward ...

What they're looking for Strong Python + data engineering fundamentals Experience building scalable data pipelines/platforms Exposure to distributed systems (Kafka, Spark, Airflow etc.) Interest in AI/ML systems is a plus ...