51 to 75 of 125 Apache Spark Jobs in the UK

Maintain the controls and records needed to support auditing, data traceability and compliance requirements. Performance and operational management Investigate and improve the performance of Spark workloads, Delta tables and SQL compute. Review resource consumption and identify ways to reduce unnecessary platform and cloud expenditure. Set up monitoring and alerting … knowledge sharing and mentoring across the team. What we are looking for Practical experience managing Databricks in a production environment. A strong understanding of Apache Spark, Delta Lake and Unity Catalog. Experience working with one or more major cloud providers, including storage, networking and identity management. Confidence using ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

inputs/outputs, and adherence to resiliency and security expectations Preferred qualifications, capabilities, and skills Experience with data lakehouse architectures using Delta Lake, Apache Spark, or similar open-source frameworks Familiarity with AWS-native data services such as Glue, Lake Formation, Redshift, or Athena Exposure to data governance ...

technical teams in data engineering, data platforms, software engineering, or AI/ML.Technical Depth: Strong technical background capable of evaluating architectures spanning distributed computing (Apache Spark), Python/Scala, modern cloud ecosystems (AWS, Azure, GCP), CI/CD, and MLOps/GenAI frameworks. Leadership Skills: Proven ability ...

PySpark, and dbt Build and optimize batch and streaming data pipelines with strong performance, fault tolerance, and observability Develop and operate workflow orchestration (e.g., Apache Airflow) to schedule, monitor, and manage data movement and transformations Model and transform data for analytics using SQL and dbt to support business intelligence … distributed data processing and performance tuning Hands-on experience with modern data warehousing/lakehouse technologies (e.g., Redshift, BigQuery, Snowflake; and engines such as Spark, Flink, or Trino; and table formats such as Iceberg, Hudi, or similar) Strong SQL skills and experience with SQL-based transformation tooling (e.g. ...

time ML & reproducibility: A strong grasp of production inference patterns and reproducible environments (Docker, MLflow or equivalent). Cloud & tooling: AWS, plus SageMaker, Snowflake, Spark/Databricks and Kubernetes. Engineering mindset: A focus on reliability, maintainability and continuous improvement. ...

into robust approaches and measurable outcomes. Strong practical capability in Python and SQL across data exploration, feature engineering, modelling and validation, with experience of Spark and modern cloud data platforms. Expertise in statistical analysis, forecasting, predictive modelling, classification, clustering and optimisation, including developing and deploying production machine-learning solutions. ...

executives, driving informed decision-making. Experience with data visualization platforms (e.g., Looker, Tableau). Familiarity with Big Data tools and platforms such as Spark and Airflow, and proficiency in version control using Git. Ability to design and develop efficient and scalable ETL and data pipelines for datasets, including ...

Proficiency in the Python data science ecosystem (e.g. NumPy, Pandas, scikit-learn, PyTorch, TensorFlow). Experience with big data and distributed analytics platforms (e.g. Spark, Hadoop). Hands-on experience with Natural Language Processing, graph analytics and advanced analytical methods. Experience using cloud analytics platforms, with strong ...

/GraphQL) Machine Learning: Foundations of MLflow, Weights & Biases, LoRA, PEFT, SageMaker, Vertex AI, AzureML, Kubeflow, Airflow, Prefect, TorchServe, Triton, vLLM, SHAP, LIME, pandas, Spark, SQLLLM tooling: Familiarity with LangChain, Semantic Kernel, prompt chaining, vector search, and context management Performance & observability: Skilled in debugging distributed systems, tuning for latency ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

Feast, AWS Feature Store) Experience with containerisation (Docker) and orchestration (Kubernetes, ECS) Familiarity with Infrastructure as Code (Terraform, CloudFormation) Experience with data processing frameworks (Spark, Dask) for large-scale workloads Understanding of data governance and compliance frameworks Experience working in regulated industries (finance, healthcare, or similar) Security Clearance ...

data extraction, transformation, and pipeline development Experience with dashboarding and visualization tools (Tableau, Qlik, or similar) Familiarity with big data tools (Snowflake, Databricks, Spark) and ETL processes Experience with Python or R for advanced analytics, automation, or experimentation support preferred Knowledge of statistical methods and experimentation (A/ ...

about you: Strong Python knowledge. Ability to read through code. Demonstrable experience collaborating with engineers. Experience with big data technologies such as Hadoop, Spark, or similar. Familiarity with cloud platforms (e.g., AWS, GCP, Azure) and data warehousing solutions. Strong knowledge in at least a few of the following areas ...

model validation. Experiment tracking and reproducibility tools, for example MLflow, Weights & Biases or equivalent. Data wrangling and analysis using Pandas, NumPy, SQL; familiarity with Spark or similar is a plus. Model optimisation and research-to-prototype deployment fundamentals, including ONNX, TorchScript, FastAPI/gRPC and GPU acceleration, with CUDA ...

robust data models, ETL pipelines, metadata frameworks, and modern data warehousing solutions. Design and optimise large-scale streaming data platforms using technologies such as Apache Flink and other real-time processing frameworks. Ensure all solutions comply with industry best practices, governance frameworks, and regulatory requirements, maintaining the highest standards … Lambda, Redshift , and related data services. Proven experience designing secure architectures within sensitive or highly regulated environments. Hands-on experience with Apache Flink and real-time data processing architectures. Strong understanding of data modelling methodologies, architectural patterns, and industry standards. Expertise across ETL processes, data integration, data warehousing, distributed ...

data models, ETL pipelines, metadata frameworks, and modern data warehousing solutions. \n Design and optimise large-scale streaming data platforms using technologies such as Apache Flink and other real-time processing frameworks. \n Ensure all solutions comply with industry best practices, governance frameworks, and regulatory requirements, maintaining the highest … Lambda, Redshift , and related data services. \n Proven experience designing secure architectures within sensitive or highly regulated environments. \n Hands-on experience with Apache Flink and real-time data processing architectures. \n Strong understanding of data modelling methodologies, architectural patterns, and industry standards. \n Expertise across ETL processes, data ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes The environment Python, SQL, Spark, Java or Scala AWS, Azure or GCP modern distributed and cloud-native architectures What we need - Active DV clearance - essential - Production-scale pipeline experience ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes **The environment** Python, SQL, Spark, Java or Scala · AWS, Azure or GCP · modern distributed and cloud-native architectures **What we need** - Active DV clearance - essential - Production-scale pipeline experience ...

senior stakeholders and contribute to business development initiatives. Provide thought leadership through industry engagement and innovation. Requirements Strong Python, SQL, Pandas, Scikit-Learn and Spark experience. Experience delivering real-world AI and machine learning systems. Understanding of statistics and machine learning methodologies. Experience with cloud platforms and large-scale ...

with experience shaping technical direction across teams and stakeholders Solid expertise in relevant technologies such as AWS, Terraform, Kafka/MSK, Airflow, Glue or Spark, and building scalable data solutions Experience with modern data ecosystems including data lakes/lakehouse architectures, Iceberg or similar table formats, as well ...

seasonality analysis) Solid foundation in statistical modelling and machine learning theory Proficiency in Python and experience working with large, complex datasets using SQL and Spark Ability to manage multiple priorities across a portfolio of projects Experience engaging both technical and business stakeholders Experience with libraries such as SciPy, Scikit ...