26 to 50 of 53 Apache Spark Jobs in the North of England

modern cloud engineering practices. Wherever you land, you'll be working with some of the biggest datasets in the UK - using everything from Spark and statistical methods to domain knowledge and emerging GenAI applications. The work you could be doing Cloud-native AI engineering and large-scale data platforms ...

support comprehensive data analysis and reporting. Implement and uphold best practices in data governance, security, and compliance, using tools like Azure Purview, UnityCatalog, and Apache Atlas tomaintaindata integrity and trust. Ensure data quality and integrity through meticulous attention to detail and rigorous QA processes, continually refining and optimising data … ease. Understanding of Data Lakehouse Architecture:You have a strong grasp of the principles and implementation of Data Lakehouse architecture. Hands-On Experience with Spark-Based Solutions:Youpossessexperience with Spark-based platforms like Azure Synapse, Databricks, Microsoft Fabric, or evenon-premiseSpark clusters, usingPySparkor Spark SQL to manage ...

problem‐solving skills to collaborate across global teams, and translate complex data insights into actionable outputs Familiarity with data platforms such as Databricks, Spark, Snowflake, or similar technologies Ability to communicate technical concepts clearly to non‐technical stakeholders Basic understanding of artificial intelligence concepts, with curiosity and enthusiasm ...

Nice to have: Proven experience leading projects and managing teams, Experience in presales, client workshops, or building Proof-of-Concepts (PoCs), Familiarity with Hadoop, Spark, Research or R&D background. Soft Skills Ability to structure projects teams, make technical decisions, and own outcomes, Ability to manage multiple projects ...

Python and SQL; APIs; test automation; data structures; distributed processing; Git; CI/CD; containers; infrastructure awareness; production debugging and performance optimization. Data engineering: Spark or Databricks, Kafka or equivalent streaming, ETL/ELT, batch and real-time pipelines, data contracts, lineage, cataloguing, quality rules, and scalable cloud storage. ...

focus on production reliability. Essential skills and experience Strong hands-on experience developing data engineering solutions using Python and PySpark. Practical experience with Apache Kafka and Apache NiFi for streaming, ingestion and orchestration. Experience working with Hadoop, Hive or comparable enterprise data-lake technologies. Solid knowledge … software development tools. Exposure to agentic AI, workflow orchestration or retrieval-augmented generation. Key responsibilities Design and develop scalable data pipelines using Python, PySpark, Apache Kafka and Apache NiFi. Integrate vendor platforms and enterprise systems through REST APIs and secure authentication mechanisms. Build data processing and storage solutions ...

strong understanding of modern data engineering best practicesYou have experience with Databricks, ideally also AzureYou have experience of working with technologies such as SQL, Apache Spark and Python including PySpark and PandasYou're excited to join a start-up in a role that you can shape and influenceYou ...

Warehouses Experience with enterprise data source integration — MS SQL Server (on-premises and Azure SQL) and Oracle Database ingestion into Azure landing zones Strong Apache Spark skills — PySpark or Scala, optimisation, partitioning, and job scheduling in Databricks Experience assessing and remediating partially built or vendor-abandoned data platform ...

Private cloud platforms (HPE Private AI, Nvidia AI Enterprise, Dell AI Factories) Understanding of data engineering practices (ETL, data lakes, big data tools like Spark). Knowledge of designing scalable, distributed systems. Familiarity with model lifecycle management and MLOps practices. Ability to align AI initiatives with business outcomes ...

Proficiency in the Python data science ecosystem (e.g. NumPy, Pandas, scikit‐learn, PyTorch, TensorFlow). Experience with big data and distributed analytics platforms (e.g. Spark, Hadoop). Hands‐on experience with Natural Language Processing, graph analytics and advanced analytical methods. Experience using cloud analytics platforms, with strong ...

maintain operational continuity and quality output Knowledge of data processing frameworks, including ETL pipelines, data integration, and data modeling (experience with Databricks or Spark preferred) Strong analytical and problem‐solving skills with the ability to communicate clearly to both technical and non‐technical stakeholders Basic understanding of artificial intelligence ...

including target modelling, cutover approach, reconciliation strategy and technical risk management. Design and build the real‐time data architecture using Amazon MSK/Apache Kafka and PySpark Structured Streaming (evaluating Apache Flink where it is the better fit), including event schema governance, exactly‐once semantics and stream … large‐scale batch and streaming processing, and strong command of data modelling and data warehousing concepts. Production experience designing and operating streaming architectures with Apache Kafka or Amazon MSK, including stream processing with Spark Structured Streaming. Hands‐on experience with open table formats, such as Apache Iceberg ...

senior stakeholders and contribute to business development initiatives. Provide thought leadership through industry engagement and innovation. Requirements Strong Python, SQL, Pandas, Scikit-Learn and Spark experience. Experience delivering real-world AI and machine learning systems. Understanding of statistics and machine learning methodologies. Experience with cloud platforms and large-scale ...

senior stakeholders and contribute to business development initiatives. Provide thought leadership through industry engagement and innovation. Requirements Strong Python, SQL, Pandas, Scikit-Learn and Spark experience. Experience delivering real-world AI and machine learning systems. Understanding of statistics and machine learning methodologies. Experience with cloud platforms and large-scale ...

demonstrable experience of working in rapidly changing client environments. Needs to be a practitioner and proponent of Agile and Dev Ops Proficiency in Hadoop, Spark, Scala, Python, or a programming language associated with data engineering Expertise in building and deploying production level data solutions maintained by application support teams ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with … platforms such as Amazon Bedrock. Experience developing data-based apps in Streamlit/Shiny Skills in ‘Big Data’ technology tools like Hadoop/Spark Knowledge of Infrastructure as Code tools like CloudFormation/Terraform If you can bring some of these skills and experiences, along with transferable strengths ...

Senior Data Engineer – SQL/Python/AWS Adria Solutions is recruiting a Senior Data Engineer for a growing business developing a new digital services platform. With established backing, the business is building its data ...

Chester (Hybrid). You will own data lifecycle, building scalable pipelines and ensuring governance in a regulated financial environment. You'll work with Python, Spark, Kafka, NiFi, Hadoop, Hive, and modern data-platform technologies, delivering production-grade controls and observability. #J-18808-Ljbffr ...

lead embedded data science teams, partnering effectively with senior stakeholders across multiple domains Hands‐on technical expertise with tools such as Databricks, Python, Spark, and GCP/BigQuery Engineering mindset, with experience moving teams toward machine learning engineering best practice Credibility to lead long‐tenured individual contributors while providing ...

deploying scalable data platforms end-to-end on AWS and/or Databricks. Production Coding: Strong, hands-on production experience with Python, SQL, Spark, and Databricks or AWS data tools. Consulting/Client Delivery: Experience working in a consultancy, systems integrator, or client-facing setting with direct accountability …/or AWS certifications at Associate or Professional level. Streaming Architectures: Practical experience with real-time or event-driven streaming (Kafka/MSK, Kinesis, Spark Structured Streaming). Advanced Data Architectures: Exposure to Data Mesh concepts, data products, or federated data governance models. AI/ML Infrastructure: Exposure ...

seeking a Secure Data Engineer in Newcastle. This role focuses on building scalable and secure data platforms for Defence clients using Azure, Databricks, and Apache Spark. The successful candidate will engage in developing robust data applications, ensuring high security and compliance standards. Opportunities for professional growth and a flexible ...

Hadoop ecosystem, collaborating with data scientists to operationalize ML models using CML. The role spans data ingestion, transformation, and full-stack tooling with Java, Spark, Python, Flask, and React, emphasizing performance tuning and efficient resource use across batch and streaming workloads. #J-18808-Ljbffr ...

include: Contributing to disaster recovery, business continuity and platform resilience solutions for streaming environments. Working with streaming technologies such as Kafka Streams, Flink or Spark Streaming and applying real‐time data processing approaches to business requirements. Working collaboratively across engineering, architecture, security and infrastructure teams, sharing knowledge and contributing ...

hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data modelling and lakehouse design (bronze, silver, gold) Terraform, CI/… pairing and design sessions Confidence communicating with technical and non-technical stakeholders, and challenging constructively Desirable: SQL Server or on-prem to cloud migration, Apache Flink, regulated industry experience If you'd rather shape a platform than inherit a finished one, and want a seat at the table while ...

hands-on AWS data experience: Glue, Lambda, S3, Redshift and EMR Advanced Python, PySpark and SQL Production experience with Kafka or Amazon MSK, including Spark Structured Streaming Hands-on experience with Apache Iceberg or Delta Lake Strong data modelling and lakehouse design (bronze, silver, gold) Terraform, CI/… pairing and design sessions Confidence communicating with technical and non-technical stakeholders, and challenging constructively Desirable: SQL Server or on-prem to cloud migration, Apache Flink, regulated industry experience If you'd rather shape a platform than inherit a finished one, and want a seat at the table while ...