1 to 25 of 26 Apache Spark Jobs in the North West

data processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex … clean, maintainable, and efficient code, ensuring adherence to best practices through code reviewsBig Data EngineeringDesign, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache AirflowOptimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performanceDevelop and fine-tune complex queries using ...

data processing. In this role, you will architect and maintain production-grade data products and APIs, leveraging technologies such as Java/Scala, SQL, Spark, Airflow, and Kubernetes to deliver reliable, high-performance solutions. You will collaborate with cross-functional partners across Finance, Accounting, and Analytics to translate complex … efficient code, ensuring adherence to best practices through code reviews Big Data Engineering Design, develop, and maintain data pipelines and ETL workflows using Apache Spark and Apache Airflow Optimise data storage, retrieval, and processing systems to ensure reliability, scalability, and performance Develop and fine-tune complex queries ...

skills, with experience of Java or Scala beneficial Experience building large-scale ETL/ELT and data processing pipelines Hands‐on experience with Apache Spark and distributed data processing technologies Experience designing and operating cloud‐native data platforms, particularly within AWS Familiarity with Docker, Kubernetes and CI/ ...

deliver agreed scope, acting as the key point of escalation for decisions that impact your area of workDemonstrable experience of building data pipelines using Spark, Python or SQLExperience working with AWS and its data and AI/ML services such as SageMakerExperience of big data and lakehouse platforms ...

e.g., AWS, Azure, GCP) and their AI/ML services.Data Engineering: Proficiency in data manipulation and analysis using SQL and big data technologies (e.g., Spark, Hadoop). Experience with ETL, data pipelines, and automated workflows for moving and processing data.Data Governance: Adherence to data governance, security, and privacy standards ...

Know: Salary: Up to £80,000 per annum Team: Work with cross-functional teams and mentor junior engineers Tools: Microsoft Fabric, Azure ML, MLflow, Spark, Python, SQL, CI/CD, and ML frameworks like TensorFlow, PyTorch, Scikit-learn Scope: End-to-end AI/ML solutions for multi-product …/ML Familiarity with Microsoft Fabric (Data Engineering, Data Science, Real-Time Analytics) and Azure ML, including MLflow Proficiency with tools such as Apache Spark, Azure Data Factory, GIT, Python and SQL Skilled in developing, training and deploying machine learning models with frameworks like TensorFlow, PyTorch or Scikit ...

volume, data-intensive environments. Extensive AWS experience including services such as Glue, Lambda, S3, Redshift, and wider cloud-native data tooling. Strong knowledge of Apache Spark and large-scale data processing. Experience working with streaming technologies such as Kafka. Expert SQL and data modelling capabilities. Experience designing modern ...

platforms for performance and reliability Collaborate with Agile teams to deliver secure, high-quality solutions Write clean, tested, maintainable code Required Skills: Experience with Apache Spark, Hadoop or similar big data technologies AWS Apache NiFi Python or Java Experience building data pipelines and ETL processes Desirable ...

platform and lakehouse transformation journeys What We're Looking For Strong commercial experience in Data Engineering Hands-on experience with Databricks and/or Apache Spark Strong PySpark and SQL development skills Experience designing, building and supporting data pipelines Experience working with Azure or AWS Good understanding ...

plusWhat it takesMust-Have Skills & ExperienceDeep expertise designing and building scalable data pipelines and data products in cloud environmentsStrong hands-on experience with Databricks, Spark, SQL, and PythonExperience developing and optimizing batch and real-time data processing solutionsStrong understanding of modern data architecture patterns, including Kimball, Data Vault, dimensional ...

time data solution using streaming ingestion and delivering real time analytics, to help our clients respond to customer behaviour as it happens using Confluent, Apache Kafka, Spark, Flink or similarAbility to conduct a full technical assessment of existing platform as-is architecture and identify the to-be landscapeProviding ...

Direct knowledge of Matillion’s Data Productivity Cloud/Matillion ETL, or primary competitive data integration platforms. Big Data & Analytics Ecosystems: Experience with Spark/PySpark, modern Data Lakes/Lakehouses, and MPP database architectures (Teradata, Vertica, Netezza). Education: Bachelor’s or Master’s degree in Computer Science ...

/ELT: Direct knowledge of Matillion’s Data Productivity Cloud/Matillion ETL, or primary competitive data integration platforms.Big Data & Analytics Ecosystems: Experience with Spark/PySpark, modern Data Lakes/Lakehouses, and MPP database architectures (Teradata, Vertica, Netezza).Education: Bachelor’s or Master’s degree in Computer Science ...

environment Experience in the design and deployment of production data pipelines from ingestion to consumption within a big data architecture, using Java, Python, Scala, Spark, SQL. Experience performing tasks such as writing scripts, extracting data using APIs, writing SQL queries etc.Experience in processing large amounts of structured and unstructured ...

problem-solving skills to collaborate across global teams and translate complex data insights into actionable outputs Familiarity with data platforms such as Databricks, Spark, Snowflake, or similar technologies Ability to communicate technical concepts clearly to non technical stakeholders Basic understanding of artificial intelligence concepts, with curiosity and enthusiasm ...

MLOps/LLMOps principles, including CI/CD for ML, model monitoring, and governance frameworks.Proficiency with large-scale data processing and storage technologies (SQL, Spark, Hadoop) is a plus.Excellent stakeholder management and communication skills, with proven ability to translate complex AI concepts for diverse audiences.You can understand client challenges ...

technologies that our Data Scientists use (we don't expect applicants to have experience with all of these):Python and Databricks for Data Science, Spark, MLFlow and Airflow for ML Workflows, Google Cloud Platform for our analytics infrastructure, dbt and BigQuery for data modelling and warehousing.Discover more about ...

depth actually gets used. AWS across the board, including S3, Glue, Redshift, Lake Formation and Kinesis Databricks and Snowflake running on top of AWS Spark and distributed processing frameworks for large-scale data Streaming technologies such as Kinesis or Kafka You're expected to pick the right tools ...

build enterprise-grade cloud data solutions, providing expert perspective on storage options (e.g., object storage, data lakes, data warehouses) and compute engine options (e.g., Spark, Databricks, Synapse Analytics, BigQuery).Implementing and optimising data lake and data warehouse solutions on cloud platforms using services such as object storage (e.g. … Databricks, AWS Glue, Google Dataflow, Azure Data Factory, or similar.Developing event-based and streaming data ingestion and processing using cloud-native PaaS services or Apache Kafka.Data ingestion, data engineering, and/or data curation using native cloud services or tools available in cloud marketplaces.Developing data governance, data cataloguing ...

lead embedded data science teams, partnering effectively with senior stakeholders across multiple domains Hands‐on technical expertise with tools such as Databricks, Python, Spark, and GCP/BigQuery Engineering mindset, with experience moving teams toward machine learning engineering best practice Credibility to lead long‐tenured individual contributors while providing ...

scripting language — Python is required.Proficiency in at least one object-oriented language.Experience with big data technologies such as HDFS, YARN, MapReduce, Hive, Kafka, Spark, Airflow, or Presto.Experience with AWS, GCP, or Looker (advantageous but not essential).Solid background in data modelling, including the design, implementation and optimisation of conceptual ...

depth actually gets used. • AWS across the board, including S3, Glue, Redshift, Lake Formation and Kinesis • Databricks and Snowflake running on top of AWS • Spark and distributed processing frameworks for large-scale data • Streaming technologies such as Kinesis or Kafka You're expected to pick the right tools ...

spans a wide range of technologies, including Java and Python-based MicroServices, AWS/GCP cloud backend systems, Big Data technologies like Hive and Spark, and modern Web applications. With a globally distributed presence across the US, India and Europe, the team thrives on collaboration, bringing together diverse perspectives … needs, entrepreneurial spiritExcellent verbal and written communication skillsBS or MS degree in Computer Science or equivalentExperience in distributed computing frameworks like - Hive/Hadoop, Apache SparkExperience in developing Finance or HR related applicationsExperience with following cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic Cache, LambdaWorking ...

address business use cases and/or to improve product/system performance, quality and accuracy.Near Real-Time and Batch Inferencing: Use infrastructure like Spark and Ray to stand up inferencing services that integrate with operational/analytics workloads.ML Infrastructure: Help build a first-class machine learning platform from … machine learning on real recommendations use cases (brownie points for productionised sequential learning use cases!).Experience with ML/distributed ML frameworks like Ray, Spark-MLlib, TensorFlow etc.Experience with real-time scoring/evaluation of models with low latency constraints.Great coding skills and strong software development experience ...

business use cases and/or to improve product/system performance, quality and accuracy. Near Real-Time and Batch Inferencing: Use infrastructure like Spark and Ray to stand up inferencing services that integrate with operational/analytics workloads. ML Infrastructure: Help build a first-class machine learning platform … learning on real recommendations use cases (brownie points for productionised sequential learning use cases!). Experience with ML/distributed ML frameworks like Ray, Spark-MLlib, TensorFlow etc. Experience with real-time scoring/evaluation of models with low latency constraints. Great coding skills and strong software development experience ...