76 to 100 of 125 Apache Spark Jobs in the UK

large language model (LLM) systems, including retrieval augmented generation (RAG) and fine-tuning. Build scalable ETL/ELT pipelines using tools such as Spark or dbt, so data is high quality, time-stamped, and ready for both training and real-time inference. Design and deploy data collection systems ...

Event Hubs on Azure, or BigQuery, Dataflow and Pub/Sub on GCP Databricks and Snowflake running on top of any of the three Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or the native cloud equivalents You’re expected to pick ...

Event Hubs on Azure, or BigQuery, Dataflow and Pub/Sub on GCP Databricks and Snowflake running on top of any of the three Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kafka or the native cloud equivalents You’re expected to pick ...

large language model (LLM) systems, including retrieval-augmented generation (RAG) and fine-tuning. Build scalable ETL/ELT pipelines using tools such as Spark or dbt, ensuring data is high quality, time-stamped and ready for both training and real-time inference. Design and deploy data collection systems ...

. Experience with Natural Language Processing (NLP), Large Language Models (LLMs) or Image Processing. Experience in areas of model development, data processing and streaming (Spark, Kafka), microservices in python (Flask or FastAPI), and interactive visualisations and User Interfaces (Streamlit, Plotly, Gradio etc). While we think the above experience ...

Familiarity with monitoring tools such as CloudWatch, Prometheus or Grafana. Useful additional experience Relevant AWS certifications. Exposure to data-processing environments using Databricks or Apache Spark. The person we are looking for You enjoy making systems easier to run and reducing the need for manual intervention. You investigate problems ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with … platforms such as Amazon Bedrock. Experience developing data-based apps in Streamlit/Shiny Skills in 'Big Data' technology tools like Hadoop/Spark Knowledge of Infrastructure as Code tools like CloudFormation/Terraform If you can bring some of these skills and experiences, along with transferable strengths ...

tasks. Key Responsibilities: Azure Platform Support & MonitoringSupport and maintain Azure-based data solutions, including: Azure Data Factory (ADF) pipelines, dataset, linked service, triggerAzure Databricks (Spark jobs, notebooks, clusters)Azure Machine Learning models and endpointsPower BI dashboards and dataset refreshesMonitor and troubleshoot failures in pipelines, jobs, and ML workflows using … Maintain detailed SOPs, runbooks, knowledge base articles, and client documentation. Required Skills and Qualifications: Azure ServicesAzure Data Factory (ADF): pipelines, triggers, parameterization, monitoringAzure Databricks: Spark, notebooks, job orchestrationAzure Machine Learning: pipelines, model deployment, monitoringPower BI Service: dataset refreshes, access control, report diagnosticsDevOps & AutomationCI/CD: Azure DevOps, GitHub Actions ...

components . Build and maintain platform capabilities supporting batch and real-time data workloads . Develop and operate distributed data services using Kafka, Kinesis, Spark, Flink or equivalent technologies . Build and maintain cloud-native data platform capabilities using AWS and/or Azure . Support data ingestion, processing … Hands-on experience with real-time streaming and event-driven architectures , including Kafka, Kinesis or equivalent. Experience with distributed data processing technologies such as Spark, Flink, Kafka Streams or equivalent. Strong experience with AWS and/or Azure cloud environments. Experience with cloud services such as S3, Glue, Lambda ...

Senior Data Engineer – SQL/Python/AWS Adria Solutions is recruiting a Senior Data Engineer for a growing business developing a new digital services platform. With established backing, the business is building its data ...

understanding of statistics, machine learning methodologies, and model evaluation. Experience delivering AI solutions from proof-of-concept through to production. Exposure to Databricks, MLflow, Spark, or similar cloud-based data platforms. Ability to influence senior stakeholders and lead technical strategy Desirable Experience Experience building multi-agent AI systems ...

Candidates Must Have: 4 years commercial experience with Linux CentOS , bare metal and virtualisation Puppet configuration management or equivalent automation experience Big data (Hadoop, Spark, Kafka) Containers and scheduling experience (Docker, LXC, Kubernetes, Mesos) Strong scripting You will be joining a progressive and exciting company committed to excellence. They ...

with demonstrable experience of working in rapidly changing client environments. Needs to be a practitioner and proponent of Agile and Dev OpsProficiency in Hadoop, Spark, Scala, Python, or a programming language associated with data engineeringExpertise in building and deploying production level data solutions maintained by application support teamsKnowledge ...

executive audiences both written and verbalPreferred (but not required) to have: Hands on experience with Python Experience working with modern data technology (e.g. dbt, spark, containers, devops tooling, orchestration tools, git, etc.) Experience with data science and machine learning technologyPeople want to buy from people who understand them. ...

. Experience with Natural Language Processing (NLP) and Large Language Models (LLMs). Experience in areas of model development, data processing and streaming (Spark, Kafka), microservices in python (Flask or FastAPI), and interactive visualisations and User Interfaces (Streamlit, Plotly, Gradio etc). These lists are not exhaustive ...

machine learning frameworks and programming languages such as Python, R, or similar. Experience working with large datasets and data processing technologies (e.g., Hadoop, Spark, SQL).Familiarity with anomaly detection, supervised, unsupervised learning methods, deep learning, and graph based solutionsDemonstrated ability to work collaboratively in cross-functional teams and effectively ...

experience Term structure modelling, stochastic calculus or Monte Carlo methods Interest rate curve bootstrapping Algorithmic execution experience Data lake or warehouse experience (Snowflake, Iceberg, Spark etc.) A product mindset - you think about who uses your systems and why. Clear communicator who can work cross-functionally with other quants, analysts ...

engineering practices (clean code, testing, version control, code review)Proficient with large-scale data: strong SQL skills and familiarity with distributed data processing (e.g., Spark, Hive) for building training datasets, features, and analytical viewsLeadership & CollaborationProven ability to lead through influence: aligning cross-functional stakeholders on problem definitions, success metrics ...

secure, scalable data solutions across the full lifecycle, including acquisition, integration, storage, processing and analysis. Develop and maintain data ingestion and transformation pipelines using Apache NiFi, Python and SQL. Design data models, schemas and data contracts/Interface Control Documents (ICDs) to support reliable integration between systems. Integrate data … take ownership of customer-facing deliverables. Strong Python and SQL skills, with experience building, testing and optimising data pipelines. Hands-on experience with Apache NiFi for data ingestion, routing, transformation and orchestration. Experience of data modelling and schema design, including defining data contracts or Interface Control Documents (ICDs). ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated ...

background)Testing and deployment pipelines, software and productivity tools, monitoring and middleware, data infrastructure. Integrating best-of-breed tools and technologies such as Spark, Kafka, Zookeeper, Mesos, Docker, ElasticsearchAssembling platforms and frameworks to automate business processesCollaborating with internal technology, research and portfolio management teamsPython Developers MUST have: A bachelor ...

following areas: software design or development, content distribution/CDN, scripting/automation, database architecture, IP networking, IT security, BigData/Hadoop/Spark, operations management, service oriented architecturePreferred qualification - Experience with AWS services or other cloud offerings- Experience with operational parameters and troubleshooting for four ...

doingDesigning and building scalable data pipelines that support machine learning, analytics and data science initiatives. Developing high-performance data solutions using Python, Spark, Databricks and Azure technologies. Creating and maintaining datasets that power customer, marketing, pricing and forecasting use cases. Collaborating closely with Applied Scientists and Machine Learning Engineers … engineering solutions within cloud-based environments. Strong expertise in Python and experience developing production-grade data pipelines and data products. Experience working with Databricks, Spark and distributed processing technologies. Experience designing and operating large-scale data platforms using Azure, AWS or GCP.Deep understanding of data modelling, data architecture ...

concisely. Good to HaveKnowledge of the adTech space and particularly the identifiers used in the field. Experience with large-scale distributed data processing (e.g., Apache Spark) and cloud data infrastructure across GCP and AWS (e.g., Dataproc, Dataflow, GKE, EMR).Experience with TemporalWe use automated decision systems ...

building and deploying machine learning models in production, alongside strong hands-on Python skills. You should be comfortable with distributed computing tools such as Spark or PySpark, cloud platforms and modern MLOps practices. Experience managing and developing data scientists is essential, as is the ability to explain technical decisions ...