76 to 100 of 138 Apache Spark Jobs in the UK excluding London

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

Warehouses Experience with enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance ...

Mathematics, or a related field. Applied experience in data science, machine learning, or related areas. Strong programming skills in Python with experience in PySpark, Spark SQL, and manipulating data. Experience LLM Orchestration and Agentic AI libraries Proficiency in using GenAI models (OpenAI or similar) to solve business problems. Experience ...

Private cloud platforms (HPE Private AI, Nvidia AI Enterprise, Dell AI Factories) Understanding of data engineering practices (ETL, data lakes, big data tools like Spark). Knowledge of designing scalable, distributed systems. Familiarity with model lifecycle management and MLOps practices. Ability to align AI initiatives with business outcomes ...

Private cloud platforms (HPE Private AI, Nvidia AI Enterprise, Dell AI Factories) Understanding of data engineering practices (ETL, data lakes, big data tools like Spark). Knowledge of designing scalable, distributed systems. Familiarity with model lifecycle management and MLOps practices. Ability to align AI initiatives with business outcomes ...

role: Python SQL Cloud platform architecture (AWS, Azure, GCP) Data Lakes/Lakehouses and analytical tools (Databricks, Azure Fabric/OneLake, AWS Lake Formation, Spark, Athena, etc.) CI/CD and other DevOps practices such as IaC Testing Additional experience with the following would be beneficial but not essential … vector databases Streaming technologies (Kafka, Kinesis, Flink, etc.) Containers and related services (Docker, Kubernetes, container Registries, etc) Data modelling approaches (Kimball, Imnon) Orchestration tools - Apache Airflow, Prefect or cloud-native tools Backend software development (Java, APIs, Scalability, Logging and Monitoring etc.) MLFlow and other MLOps/Machine Learning Engineering ...

services, with secure integration of enterprise data sources. AWS engineering experienceincluding containers, Kubernetes, CI/CD , a pplication resilience and security . Experience with Apache Spark or similar data-processing frameworks and large structured datasets. Sound judgement when using AI-assisted development tools , including validation of outputs ...

into robust approaches and measurable outcomes. Strong practical capability in Python and SQL across data exploration, feature engineering, modelling and validation, with experience of Spark and modern cloud data platforms. Expertise in statistical analysis, forecasting, predictive modelling, classification, clustering and optimisation, including developing and deploying production machine-learning solutions. ...

approaches Experience with Generative AI techniques such as retrieval-augmented generation, fine-tuning, or prompt engineering Exposure to distributed data processing frameworks, such as Spark Exposure to fraud detection, anti-money laundering, anomaly detection, or graph and network-based modelling Awareness of model risk management, governance, and regulatory controls ...

Proficiency in the Python data science ecosystem (e.g. NumPy, Pandas, scikit‐learn, PyTorch, TensorFlow). Experience with big data and distributed analytics platforms (e.g. Spark, Hadoop). Hands‐on experience with Natural Language Processing, graph analytics and advanced analytical methods. Experience using cloud analytics platforms, with strong ...

Proficiency in the Python data science ecosystem (e.g. NumPy, Pandas, scikit‐learn, PyTorch, TensorFlow). Experience with big data and distributed analytics platforms (e.g. Spark, Hadoop). Hands‐on experience with Natural Language Processing, graph analytics and advanced analytical methods. Experience using cloud analytics platforms, with strong ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

edge and implementing secure configuration distribution Building streaming ingestion, event pipelines, batch/stream processing workloads using both distributed event streaming technologies and Apache Flink and/or Apache Spark with clear SLA’s and repeatable deployment Implementing metadata, lineage and catalogue capabilities and implementing policy-based ...

design artefacts and delivery roadmaps Key Requirements 7+ yearsâ experience in data engineering or data platform development Strong experience with modern cloud data platforms, Spark and distributed data processing Experience with data cataloguing and governance tooling, open table formats (e.g. Delta Lake, Iceberg) and streaming architectures Advanced Python ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

maintain operational continuity and quality output Knowledge of data processing frameworks, including ETL pipelines, data integration, and data modeling (experience with Databricks or Spark preferred) Strong analytical and problem‐solving skills with the ability to communicate clearly to both technical and non‐technical stakeholders Basic understanding of artificial intelligence ...

Scale processing across distributed cloud warehouses/storage via container orchestration (e.g. Kubernetes), High-Performance GPU Compute (e.g. Slurm), or distributed compute frameworks (e.g. Spark, Ray). Contribute to an engineering culture that values maintainability, testing, robust system design, and deep collaboration, but allows flexibility for rapid prototyping ...

model validation. Experiment tracking and reproducibility tools, for example MLflow, Weights & Biases or equivalent. Data wrangling and analysis using Pandas, NumPy, SQL; familiarity with Spark or similar is a plus. Model optimisation and research-to-prototype deployment fundamentals, including ONNX, TorchScript, FastAPI/gRPC and GPU acceleration, with CUDA ...

including target modelling, cutover approach, reconciliation strategy and technical risk management. Design and build the real‐time data architecture using Amazon MSK/Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly‐once semantics and stream … large‐scale batch and streaming processing, and strong command of data modelling and data warehousing concepts. Production experience designing and operating streaming architectures with Apache Kafka or Amazon MSK, including stream processing with Spark Structured Streaming. Hands‐on experience with open table formats, such as Apache Iceberg ...

senior stakeholders and contribute to business development initiatives. Provide thought leadership through industry engagement and innovation. Requirements Strong Python, SQL, Pandas, Scikit-Learn and Spark experience. Experience delivering real-world AI and machine learning systems. Understanding of statistics and machine learning methodologies. Experience with cloud platforms and large-scale ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes **The environment** Python, SQL, Spark, Java or Scala · AWS, Azure or GCP · modern distributed and cloud-native architectures **What we need** - Active DV clearance - essential - Production-scale pipeline experience ...

senior stakeholders and contribute to business development initiatives. Provide thought leadership through industry engagement and innovation. Requirements Strong Python, SQL, Pandas, Scikit-Learn and Spark experience. Experience delivering real-world AI and machine learning systems. Understanding of statistics and machine learning methodologies. Experience with cloud platforms and large-scale ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...