1 to 25 of 140 Apache Spark Jobs in the UK

systems are developed and used responsibly and ethically. Tooling the Future: Get hands-on with cutting-edge technologies like Hugging Face, PyTorch, TensorFlow, Apache Spark, Apache Airflow, and other modern data and ML frameworks. Collaborate and Lead: Partner closely with ML Engineers, Data Scientists, and Researchers … services. Programming Prowess: Strong programming skills in Python and SQL are essential. Big Data Ecosystem Expertise: Hands-on experience with big data technologies like Apache Spark, Kafka, and data orchestration tools such as Apache Airflow or Prefect. ML Data Acumen: Solid understanding of data requirements for machine ...

Data Engineer Salary up to £60,000 + discretionary bonus Hybrid working model with collaborative office environment Work with Databricks, Spark Structured Streaming, Kafka (MSK), and AWS Apply online or contact Chelsea Hackett via chelsea.hackett@searchability.com ABOUT THE CLIENT Due to continued growth, we’re seeking a skilled Data … optimise scalable data pipelines within a Databricks Lakehouse environment. You’ll work with both streaming and batch data pipelines using technologies such as Apache Spark Structured Streaming, Delta Live Tables, and Kafka, ensuring data flows efficiently and reliably across the organisation. Working closely with BI, product, and engineering ...

capabilities across the organization. Key Responsibilities Design and implement data architectures on the Databricks Lakehouse Platform Build and optimize ETL/ELT pipelines using Apache Spark and Databricks Implement Delta Lake and medallion architecture (Bronze, Silver, Gold) Ensure data governance, security, and performance optimization Integrate Databricks with cloud … stakeholders to deliver scalable data solutions Required Skills & Qualifications Databricks Certification (e.g., Databricks Certified Data Engineer Associate/Professional) Strong experience with Databricks, Apache Spark, and Delta Lake Proficiency in Python, SQL, and data pipeline development Experience with cloud platforms (Azure, AWS, or GCP) Solid understanding of modern ...

help other members of the team. · Experience provisioning and managing AWS services using infrastructure as code such as AWS CloudFormation or Terraform. · Understanding of Apache Spark, Apache Trino or other big data and machine learning systems. · Knowledge of Serverless technologies, framework and best practices. · Experience provisioning … managing Apache Airflow. · Exposure to using AI tools to enhance productivity and quality. Why Join RealityMine? At RealityMine, we believe our people are at the heart of everything we do. That’s why we go the extra mile to support every team member to unlock their full potential. Whether ...

code and sharing knowledge across the team Supporting production systems when needed The platform uses a mix of modern data tooling including AWS and Apache Spark, so there's plenty of opportunity to work with big data technologies. What they're hoping you'll bring … complex data problems An interest in improving pipelines, tooling and processes A collaborative approach and willingness to share ideas Exposure to tools like Apache Spark, modern data platforms, or AI tools for development productivity would be a bonus but isn't essential. The setup The team works ...

code and sharing knowledge across the team Supporting production systems when needed The platform uses a mix of modern data tooling including AWS and Apache Spark, so there's plenty of opportunity to work with big data technologies. What they're hoping you'll bring … complex data problems An interest in improving pipelines, tooling and processes A collaborative approach and willingness to share ideas Exposure to tools like Apache Spark, modern data platforms, or AI tools for development productivity would be a bonus but isn't essential. The setup The team works ...

with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python Experience within the following technologies is desirable - Apache Spark, Apache Airflow, Kafka, Elasticsearch/Open Search, PostgreSQL, MongoDB, Pandas, Polars About the Package £100k - £110k on the base ...

acceptable). Experience with NoSQL databases. Experience with vector databases used in modern AI systems. Strong programming experience in Python. Frameworks/Infrastructure:- Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch ...

flexibility to attend key meetings onsite. Required: Good knowledge and experience using programming languages such as Python, Java or SQL and tools such as Apache Spark or in this instance LakeFS, for data processing and management. Useful: R, Apache Spark Very Useful: Python, SQL, JSON, LakeFS ...

direction within your team Required Skills and Experience Strong analytic skills related to working with unstructured datasets Python (PySpark, Pandas, PyArrow) Distributed data processing (Apache Spark) Data ETL (Apache Airflow, AWS Step Functions, Apache NiFi) Cloud services (AWS, Azure or GCP) Messaging/Streaming (Kafka ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments Experience implementing and configuring training simulation ...

with Red Hat OpenShift and Ceph storage platforms Programming experience (e.g. Python, JavaScript) Experience working with event streaming and data pipeline technologies (e.g. Kafka, Apache NiFi, Apache Spark) Experience delivering multi-system integration projects within defence, government, or highly regulated environments. Experience implementing and configuring training simulation ...

warehousing, and data governance frameworks. Define and enforce best practices, security standards, and performance optimization across Databricks workloads. Design scalable big data architectures leveraging Spark, Delta Live Tables, Unity Catalog , and MLflow. Lead cloud architecture design on Azure/AWS/GCP (based on organization need) integrated with Databricks. … years overall experience in data engineering or data architecture. 5+ years of hands-on architecture experience specifically in Databricks . Expert in Apache Spark , PySpark , Databricks SQL , Delta Lake, and distributed systems. Strong understanding of Lakehouse architecture , BI ecosystems, and modern data platforms. Cloud expertise in Azure/ ...

logic into high-quality code using Python, Scala, or Java. Innovation: Evaluate and implement new technologies across Cloud (Azure, AWS, GCP) and Big Data (Spark, Kafka) stacks. Collaboration: Work closely with business teams to define data flows and ensure security protocols (encryption/authentication) are met. Technical Toolkit Languages ...

Time series, NLP, recommendation systems (as applicable) Programming Python (NumPy, Pandas, Scikit learn) Optional: R, SQL Data Platforms Relational & NoSQL databases Big data frameworks (Spark, Hive, Databricks) MLOps & Deployment Model lifecycle management CI/CD pipelines Containerization (Docker, Kubernetes desirable) Model packaging and REST APIs Cloud & Tools (Any combination ...

Production Hands-On Experience with Deep Learning Frameworks (TensorFlow, PyTorch, XGBoost) Deep Knowledge of Data Warehousing, ETL/ELT Architectures & Heavy Analytic Environments (Apache Beam, Hadoop, Spark, Hive) Responsibilities: Leading Customer-Facing AI Deployments across Europe, from the Initial Consulting Phase right through to the Live Implementation … Production ML Systems Architecting & Optimising Scalable Data Pipelines, ETL/ELT Solutions & Distributed ML Environments (using Hadoop, Spark & Apache Beam). Deploying Recommendation Engines & Deep Learning Models (TensorFlow, PyTorch) to solve specific Client Problems. Running Technical Design Sessions, Tailoring ML Ops Architectures using Vertex AI & Gen AI. Writing ...

data estate. Deep expertise in Microsoft Azure cloud services - ADF, ADLS, Synapse, Purview. Databricks - Delta Lake architecture, optimisation and advanced data processing. Apache Spark for large-scale distributed computing and performance tuning. Microsoft Fabric - OneLake and DirectLake integration. Azure Synapse Analytics for enterprise-scale data warehousing. Infrastructure ...

analytics environments. Knowledge of additional AWS services such as S3, Lambda, Athena, Kinesis, EMR, Fargate, or API Gateway. Experience with big data technologies including Spark, Databricks, Hadoop, or Dask. Familiarity with data visualisation tooling such as Plotly. Experience mentoring junior engineers or leading technical initiatives. What You’ll Receive ...

schema). Version Control Practical experience with Git (branching, merging, pull requests). Preferred Qualifications (A Plus) Experience with a distributed computing framework like Apache Spark (using PySpark). Familiarity with cloud data services ( AWS S3/Redshift, Azure Data Lake/Synapse, or Google BigQuery/Cloud … Storage ). Exposure to workflow orchestration tools ( Apache Airflow, Prefect, or Dagster ). Bachelor's degree in Computer Science, Engineering, Information Technology, or a related field. ...

pragmatically. Data product-oriented design . Strong judgment on when traditional approaches still make sense. Data Engineering & Platforms Strong understanding of distributed processing (e.g. Apache Spark). Familiarity with the Hadoop ecosystem . Working knowledge of SQL and NoSQL technologies. Understanding of ETL vs ELT trade-offs . ...

pragmatically. Data product-oriented design . Strong judgement on when traditional approaches still make sense. Data Engineering & Platforms Strong understanding of distributed processing (e.g. Apache Spark). Familiarity with the Hadoop ecosystem . Working knowledge of SQL and NoSQL technologies. Understanding of ETL vs ELT trade-offs . ...

/CD platform (Github Actions & Admin), Password/Secret management (hashicorp vault). Strong Data related programming skills SQL/Python/Spark/Scala. Database technologies in relation to Data Warehousing/Data Lake/Lake housing patterns and relevant experience when handling structured and non-structured data ...

within scalable cloud -based architectures. Robust understanding of data governance, data quality, and metadata management. Desirable skills Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience in using Git, Azure DevOps, or GitHub Actions for version control, CI/ ...

within scalable cloud -based architectures. Robust understanding of data governance, data quality, and metadata management. Desirable skills Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience in using Git, Azure DevOps, or GitHub Actions for version control, CI/ ...