1 to 25 of 31 Permanent Apache Spark Jobs in the North West

culture. Key Responsibilities Design, build, and maintain scalable data solutions to support business objectives. Work with Microsoft Fabric to develop robust data pipelines. Utilise Apache Spark and the Spark API to handle large-scale data processing. Contribute to data strategy, governance, and architecture best practices. Identify and … approaches. Collaborate with cross-functional teams to deliver projects on time . Key Requirements ? Hands-on experience with Microsoft Fabric . ? Strong expertise in Apache Spark and Spark API . ? Knowledge of data architecture, engineering best practices, and governance . ? DP-600 & DP-700 certifications are highly More ❯

you be doing? Design and implement efficient ETL processes for data extraction, transformation, and loading. Build real-time data processing pipelines using platforms like Apache Kafka or cloud-native tools. Optimize batch processing workflows with tools like Apache Spark and Flink for scalable performance. Infrastructure Automation: Implement … Integrate cloud-based data services with data lakes and warehouses. Build and automate CI/CD pipelines with Jenkins, GitLab CI/CD, or Apache Airflow. Develop automated test suites for data pipelines, ensuring data quality and transformation integrity. Monitoring & Performance Optimization: Monitor data pipelines with tools like Prometheus More ❯

IaC), automation & configuration management; • Ansible (plus Puppet, Saltstack), Terraform, CloudFormation; • NodeJS, REACT/MaterialUI (plus Angular), Python, JavaScript; • Big data processing and analysis, e.g. Apache Hadoop (CDH), Apache Spark; • RedHat Enterprise Linux, CentOS, Debian or Ubuntu. • Java 8, Spring framework (preferably Spring boot), AMQP - RabbitMQ, • Open source More ❯

problem-solving skills Strong teamwork, interpersonal and collaboration skills with colleagues and clients Desirable: Experience with Cloud ETL tools such as Databricks/Snowflake, Spark and Kafka Experience using source control tools such as GitHub or Azure DevOps Experience with Azure DevOps for CI/CD pipeline development and More ❯

architecture, data modelling, ETL/ELT processesand data pipeline development. Competency with cloud platforms (e.g., AWS, Azure, GCP) and big data technologies (e.g., Hadoop, Spark, Kafka etc). Excellent communication and leadership skills, with the ability to engage and influence stakeholders at all levels. Insightful problem-solving skills and More ❯

architecture, data modelling, ETL/ELT processesand data pipeline development. Competency with cloud platforms (e.g., AWS, Azure, GCP) and big data technologies (e.g., Hadoop, Spark, Kafka etc). Excellent communication and leadership skills, with the ability to engage and influence stakeholders at all levels. Insightful problem-solving skills and More ❯

Knutsford Contract Role Job Description: AWS Services: Glue, Lambda, IAM, Service Catalogue, Cloud Formation, Lake Formation, SNS, SQS, Event Bridge Language & Scripting: Python and Spark ETL: DBT Good to Have: Airflow, Snowflake, Big Data (Hadoop), and Teradata Responsibilities: Serve as the primary point of contact for all AWS related More ❯

high-impact analytics and machine learning solutions. Key Responsibilities Engineer and maintain modern data platforms with a strong focus on Databricks, including Delta Lake, Apache Spark, and MLflow Build and optimise CI/CD pipelines, infrastructure-as-code (IaC), and cloud integrations (Azure preferred, AWS/GCP beneficial … Qualifications 7+ years of experience in platform engineering, DevOps, or cloud infrastructure, with a focus on data platforms Advanced hands-on experience with Databricks, Spark, Delta Lake, and Python Proficient with Azure services (Data Factory, Storage, DevOps) and experience with IaC tools (Terraform, Bicep, ARM) Experience supporting data pipelines More ❯

Cassandra, and Redis. In-depth knowledge of ETL/ELT pipelines, data transformation, and storage optimization. Skilled in working with big data frameworks like Spark, Flink, and Druid. Hands-on experience with both bare metal and AWS environments. Strong programming skills in Python, Java, and other relevant languages. Proficiency More ❯

related field. - Proven experience (5+ years) in developing and deploying data engineering pipelines and products - Strong proficiency in Python - Experienced in Hadoop, Kafka or Spark - Experience leading/mentoring junior team members - Strong communication and interpersonal skills, with the ability to effectively communicate complex technical concepts to both technical More ❯

with innovative ideas or coding challenges. Familiarity with Agile practices in a collaborative team setting. Exposure to big data tools, such as Hadoop and Spark, for managing large-scale datasets. Experience with cloud platforms like Microsoft Azure. Why Join? They have created a super collaborative culture offering a great More ❯

engineering teams in a consultancy or enterprise setting. Deep expertise in cloud platforms (AWS & Azure), data warehousing (Snowflake, BigQuery, Redshift), and big data technologies (Spark, Hadoop). Strong background in software engineering practices, CI/CD, and infrastructure-as-code. Excellent stakeholder management and communication skills. Strategic mindset with More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

Terraform experience. Strong Kubernetes experience, we want to see people who have worked on Kubernetes implementation, involved in architecture, hands on migration etc K8S, Spark S3 Engine Terraform Ansible CI/CD Hadoop Linux/RHEL - on prem background/container management Grafanna or Elastic Search- for observability Kubernetes More ❯

analytics or BI role Strong skills in Power BI, SQL, and data modelling Experience with modern cloud and data technologies: Azure, Fabric, AWS, Databricks, Spark Understanding of machine learning, predictive analytics, and AI platforms Excellent stakeholder management and communication skills Comfortable operating independently and driving change A strategic thinker More ❯

one or more of the following would also be beneficial : Scala AWS services like Kinesis, RDS, Elasticache, S3, Athena, Data pipeline, Glue, Lambda, EMR, Spark, EC2, ECS, CloudWatch or Elastic Beanstalk Jenkins or similar tools would also be a plus A team player - we highly regard collaboration. Knowledge or More ❯

s degree/PhD in Computer Science, Machine Learning, Applied Statistics, Physics, Engineering or related field Strong mathematical and statistical skills Experience with Python, Spark and SQL Experience implementing and validating a range of machine learning and optimization techniques Effective scientific communication for varied audiences Autonomy and ownership of More ❯

s degree/PhD in Computer Science, Machine Learning, Applied Statistics, Physics, Engineering or related field Strong mathematical and statistical skills Experience with Python, Spark, and SQL Experience implementing and validating a range of machine learning and optimization techniques Effective scientific communication for varied audiences Autonomy and ownership of More ❯

GCP). Proficiency in SQL and experience with relational databases such as MySQL, PostgreSQL, or Oracle. Experience with big data technologies such as Hadoop, Spark, or Hive. Familiarity with data warehousing and ETL tools such as Amazon Redshift, Google BigQuery, or Apache Airflow. Proficiency in at least one More ❯

Our systems are self-healing, responding gracefully to extreme loads or unexpected input. We use modern languages like Kotlin and Scala, Data technologies Kafka, Spark, MLflow, Kubeflow, VastStorage, StarRocks and agile development practices. Most importantly, we hire great people from around the world and empower them to be successful. More ❯