26 to 50 of 116 Apache Spark Jobs in the UK

Knowledge of data modelling, data warehousing and Lakehouse architectures . Experience with data quality, governance and CI/CD pipelines . Exposure to Kafka, Spark Streaming or real-time data processing would be advantageous. Requirements: Ideally 7+ years of commercial Data Engineering experience. Strong hands-on technical background ...

need as Data Engineer: Passion for building scalable, resilient cloud data platforms Strong experience with Databricks or Snowflake on AWS Proven Python skills, including Spark and Airflow expertise Advanced SQL skills and end-to-end data modelling experience Experience building batch and real-time data integrations Hands-on CICD ...

with the modern data stack, building data ingestion pipelines and working with ETL and orchestration tools (e.g., Airflow, Luigi, Argo, dbt), big data technologies (Spark, Kafka, Parquet), and web frameworks for model serving (e.g. Flask or FastAPI).Data Science: Familiarity or experience with classical NLP techniques (BERT, topic modelling ...

ideal candidate will have: * Experience in data engineering, ideally aerospace/defence or industrial sectors. * ETL/ELT tools (Airflow, Azure, Microsoft SSIS), streaming (Spark, Kafka). * Strong Python and SQL; C++, C#, or MATLAB a plus. * Databases: PostgreSQL, SQL Server, Oracle, Snowflake, Iceberg, TimescaleDB. * Data Platform Capabilities ...

GCP. MSc or equivalent experience in Data Science, Statistics, Mathematics, Computer Science, Artificial Intelligence, Physics, Engineering or a related quantitative discipline. Experience with Spark or distributed data processing frameworks. Experience preparing data pipelines or analytical datasets to support machine learning workloads. Awareness of data engineering, data architecture, data governance ...

data analytics, or data science, with the ability to work effectively with various data types and sources. Experience using big data technologies (e.g. Hadoop, Spark, Hive) and database management systems (e.g. SQL and NoSQL).Graph Database Expertise: Deep understanding of graph database concepts, data modeling, and query languages (e.g. ...

Skills & Experience Strong commercial experience in data engineering within cloud-based environments. Advanced Python and SQL development skills. Hands-on experience with Azure Databricks, Apache Spark and PySpark. Good understanding of Delta Lake, lakehouse architecture and modern data modelling. Experience building and managing production data pipelines, including batch ...

skills Experience building data pipelines and integrating data from multiple sources Exposure to technologies such as Power BI, Snowflake, SQL Server, PostgreSQL, Spark, Kafka, Airflow, Docker, and Kubernetes Experience within engineering, manufacturing, industrial, or other complex regulated environments would be advantageous Excellent stakeholder management and problem-solving abilities What ...

Server, Oracle, Snowflake, Databricks).Experience with data integration (ETL/ELT, APIs).Familiarity with cloud data platforms (AWS, Azure, GCP) and big data (Hadoop, Spark, Kafka) a plus. Domain Knowledge: Exceptional understanding of the Funds business lifecycle, financial instruments, market data, and investment operations. In-depth knowledge of regulatory ...

Lambda, ECS, RDS, S3, SQS, API Gateway, and CloudWatch. Design, develop, and maintain data pipelines and ETL workflows using technologies such as Glue, Spark/PySpark, Kafka, or similar platforms. Implement and maintain Infrastructure as Code (IaC) using Terraform or equivalent tools. Drive observability, monitoring, logging, and operational readiness ...

requiredDo not rule yourself out if you do not have these. They are a bonus, not a checklist. Experience with Delta Lake, Iceberg, Spark or Databricks, PalantirExperience optimising LLM serving with tools such as vLLM, TGI or TensorRT LLMSearch and ranking experience. For example Elasticsearch or rerankersBackground in time ...

into business processes and decision-making. Technical Skills: Hands-on experience in one or more of the following areas: Programming & Analytics – Python, SQL, R, Spark, machine learning and advanced analytics Cloud & Data Platforms – Microsoft Fabric, Databricks, Snowflake, Azure, AWS and GCP AI Technologies – GenAI frameworks, Agentic AI, vector databases ...

migration strategies as well as with machine learning, LLMs, and agentic AI workflows. Proficient in programming, debugging, and problem-solving using SQL, Python, Apache Spark, cloud platforms, and Lakehouse architectures and with AI tools. Experience collaborating with Global System Integrators (GSIs) and third-party consulting organizations to drive ...

Ingestion projects in Azure environment. Demonstrated analytical and problem-solving skills particularly those that apply to a big data environment. Experience on Python scripting, Spark SQL PySpark is a plus. We use generative AI tools to support our candidate screening process. This helps us ensure a fair, consistent ...

production rather than for novelty. You don't need to know all of it on day one. Data and compute: S3, Athena, Spark on EMR, SQLOrchestration and MLOps: Airflow, MLflow, Docker, ECS, Terraform, GitHub Actions for CI/CDModels: foundation models served through Amazon BedrockAgents: LangGraph for orchestration, deployed ...

experience building, deploying, and scaling AI/ML solutions using frameworks and platforms such as Scikit-learn, PyTorch, TensorFlow, Hugging Face, LangChain, Databricks, Spark, vector databases, cloud services (AWS/Azure), and MLOps tools including MLflow and Kubeflow. Proven ability to develop production-grade, large-scale AI systems with ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable."Nice to Have"Azure, Databricks, Apache Spark or comparable cloud and data platforms. Data pipelines, ETL and workflow-orchestration technologies. Machine-learning methods such as forecasting, prediction or classification. ...

experience with the SQL Server platform and T-SQL querying. Familiarity with cloud data platforms (e.g., AWS, Azure) and big data technologies (e.g., Kafka, Spark, Databricks, Snowflake).Expert-level knowledge of building ETL processes in Python. Expert-level knowledge of performance tuning and optimization in a Python/ ...

skills in Python, and experience with Java or Scala. Data Engineering at Scale: Solid ELT/ETL design experience and hands-on work with Apache Spark. Bonus points if you have: Multi-Cloud Exposure: Experience with Azure alongside GCP.Container Orchestration: Experience packaging and deploying workloads with Helm and Kubernetes. ...

degree, or experience in a professional field or military- Experience as technical specialist in design and architecture- Experience in database (eg. SQL, NoSQL, Hadoop, Spark, Kafka, Kinesis)- Experience in consulting, design and implementation of serverless distributed solutions- Experience in software development with object oriented language- Experience in cloud based ...

Applying advanced techniques in LLMs, agentic systems, vector databases, fine-tuning, and deployment tools (e.g., HuggingFace, Langchain).Hands-on experience working with Distributed Spark based systemsExperience with data engineering concepts or a good understanding of data engineering concepts Pre-sales or post-sales experience working with external clients across … variety of industry markets. Minimum of 5+ years of customer-facing experience would be preferred[Preferred] Experience working with Apache Spark to process large-scale distributed datasetsCommunication: Proven ability to communicate and teach complex technical concepts to both technical and non-technical audiences. Core Traits: Passion for lifelong ...

Maintain the controls and records needed to support auditing, data traceability and compliance requirements. Performance and operational management Investigate and improve the performance of Spark workloads, Delta tables and SQL compute. Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure. Set up monitoring and alerting … knowledge sharing and mentoring across the team. What we are looking for Practical experience managing Databricks in a production environment. A strong understanding of Apache Spark, Delta Lake and Unity Catalog. Experience working with one or more major cloud providers, including storage, networking and identity management. Confidence using ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...