26 to 50 of 1,250 Apache Spark Jobs in England

Qualifications 5+ years of experience in data engineering roles with progressively increasing responsibility Proven experience designing and implementing complex data pipelines at scale Strong knowledge of distributed computing frameworks (Spark, Hadoop ecosystem) Experience with cloud-based data platforms (AWS, Azure, GCP) Proficiency in data orchestration tools (Airflow, Prefect, Dagster, or similar) Solid programming skills in Python, Scala, or Java More ❯

Python, Scala, or Java Experience with cloud data platforms such as GCP (BigQuery, Dataflow) or Azure (Data Factory, Synapse) Expert in SQL, MongoDB and distributed data systems such as Spark, Databricks or Kafka Familiarity with data warehousing concepts and tools (e.g. Snowflake) Experience with CI/CD pipelines, containerization (Docker), and infrastructure-as-code (Terraform, CloudFormation) Strong understanding of More ❯

Python, Scala, or Java Experience with cloud data platforms such as GCP (BigQuery, Dataflow) or Azure (Data Factory, Synapse) Expert in SQL, MongoDB and distributed data systems such as Spark, Databricks or Kafka Familiarity with data warehousing concepts and tools (e.g. Snowflake) Experience with CI/CD pipelines, containerization (Docker), and infrastructure-as-code (Terraform, CloudFormation) Strong understanding of More ❯

maintain the data platform. You will be working on complex data problems in a challenging and fun environment, using some of the latest Big Data open-source technologies like Apache Spark, as well as Amazon Web Service technologies including Elastic MapReduce, Athena and Lambda to develop scalable data solutions. Key Responsibilities: Adhering to Company Policies and Procedures with … and specifications. Good interpersonal skills, positive attitude, willing to help other members of the team. Experience debugging and dealing with failures on business-critical systems. Preferred Qualifications: Exposure to Apache Spark, Apache Trino, or another big data processing system. Knowledge of streaming data principles and best practices. Understanding of database technologies and standards. Experience working on large More ❯

Please speak to us if you have ..... .....the following professional aspirations Skill Enhancement: Aspires to deepen technical expertise in data engineering practices, including mastering tools and technologies like Apache Spark, Kafka, cloud platforms (AWS, Azure, Google Cloud), and data warehousing solutions. Career Progression : Aims to advance to a senior data engineer or data architect role, with long … Redshift, Google BigQuery, Snowflake, or Azure Synapse Analytics, including data modelling and ETL processes. ETL Processes: Proficient in designing and implementing ETL (Extract, Transform, Load) processes using tools like Apache NiFi, Talend, or custom scripts. Familiarity with ELT (Extract, Load, Transform) processes is a plus. Big Data Technologies : Familiarity with big data frameworks such as Apache Hadoop and … Apache Spark, including experience with distributed computing and data processing. Cloud Platforms: Proficient in using cloud platforms (e.g., AWS, Google Cloud Platform, Microsoft Azure) for data storage, processing, and deployment of data solutions. Data Pipeline Orchestration : Experience with workflow orchestration tools such as Apache Airflow or Prefect to manage and schedule data pipelines. Data Modelling : Strong understanding More ❯

Automate data quality and observability checks, proactively surfacing issues. Qualifications We seek experienced Data Engineers passionate about data, eager to implement best practices in a dynamic environment. Proficiency in Spark/PySpark, Azure data technologies, Python or Scala, SQL. Experience with testing frameworks like pytest or ScalaTest. Knowledge of open table formats such as Delta, Iceberg, or Apache … and agile methodologies like Scrum or Kanban. Nice to have skills: Experience in retail or e-commerce. Knowledge of Big Data and Distributed Computing. Familiarity with streaming technologies like Spark Structured Streaming or Apache Flink. Additional programming skills in PowerShell or Bash. Understanding of Databricks Ecosystem components. Experience with Data Observability or Data Quality frameworks. Additional Information What More ❯

technical and professional experience Preferred Skills: Experience working within the public sector. Knowledge of cloud platforms (e.g., IBM Cloud, AWS, Azure). Familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop). Understanding of data warehousing concepts and experience with tools like IBM Cognos or Tableau. Certifications:While not required, the following certifications would be highly beneficial … Experience working within the public sector. Knowledge of cloud platforms (e.g., IBM Cloud, AWS, Azure). Familiarity with big data processing frameworks (e.g., Apache Spark, Hadoop). Understanding of data warehousing concepts and experience with tools like IBM Cognos or Tableau. ABOUT BUSINESS UNIT IBM Consulting is IBM's consulting and global professional services business, with market leading More ❯

AWS or Azure) for deploying and scaling data systems. Highly desired experience with Azure, particularly Lakehouse and Eventhouse architectures. Experience with relevant infrastructure and tools including NATS, Power BI, Apache Spark/Databricks, and PySpark. Hands-on experience with data warehousing methodologies and optimization libraries (e.g., OR-Tools). Experience with log analysis, forensic debugging, and system performance More ❯

industries) on some of our most complex projects - individually or by leading small delivery teams. Our projects are fast-paced, typically 2 to 4 months long, and primarily use Apache Spark/Databricks on AWS/Azure. You will manage customer relationships either alone or with a Project Manager, and support our pre-sales, mentoring, and hiring efforts. More ❯

have framework experience within either Flask, Tornado or Django, Docker Experience working with ETL pipelines is desirable e.g. Luigi, Airflow or Argo Experience with big data technologies, such as Apache Spark, Hadoop, Kafka, etc. Data acquisition and development of data sets and improving data quality Preparing data for predictive and prescriptive modelling Hands on coding experience, such as More ❯

data models/schemas and developing ETL/ELT scripts in large organisations or data platforms. Strong experience leading teams and managing individuals. Strong Python and other programming skills (Spark/Scala desirable). Experience both using and building APIs. Strong SQL background. Exposure to big data technologies (Spark, Hadoop, Presto, etc.). Works well collaboratively, and independently More ❯

science use-cases across various industries Design and develop feature engineering pipelines, build ML & AI infrastructure, deploy models, and orchestrate advanced analytical insights Write code in SQL, Python, and Spark following software engineering best practices Collaborate with stakeholders and customers to ensure successful project delivery Who we are looking for We are looking for collaborative individuals who want to More ❯

dialect of data. Libraries Tools : Terraform, Flask, Pandas, FastAPI, Dagster, GraphQL, SQLAlchemy, GitLab, Athena. Your Trusted Companions : Docker, Snowflake, MongoDB, Relational Databases (eg MySQL, PostgreSQL), Dagster, Airflow/Luigi, Spark, Kubernetes. Your AWS Kingdom : Lambda, Redshift, EC2, ELB, IAM, RDS, Route53, S3-the building blocks of cloud mastery. Your Philosophy : Continuous integration/deployments, (CI/CD) automation, rigorous More ❯

production issues. Optimize applications for performance and responsiveness. Stay Up to Date with Technology: Keep yourself and the team updated on the latest Python technologies, frameworks, and tools like Apache Spark , Databricks , Apache Pulsar , Apache Airflow , Temporal , and Apache Flink , sharing knowledge and suggesting improvements. Documentation: Contribute to clear and concise documentation for software, processes … cloud platforms like AWS , GCP , or Azure . DevOps Tools: Familiarity with containerization ( Docker ) and infrastructure automation tools like Terraform or Ansible . Real-time Data Streaming: Experience with Apache Pulsar or similar systems for real-time messaging and stream processing is a plus. Data Engineering: Experience with Apache Spark , Databricks , or similar big data platforms for … processing large datasets, building data pipelines, and machine learning workflows. Workflow Orchestration: Familiarity with tools like Apache Airflow or Temporal for managing workflows and scheduling jobs in distributed systems. Stream Processing: Experience with Apache Flink or other stream processing frameworks is a plus. Desired Skills: Asynchronous Programming: Familiarity with asynchronous programming tools like Celery or asyncio . Frontend More ❯

code to ensure high availability and accessibility Requirements Minimum of 8 years of experience in data engineering At least 5 years of hands-on experience with Azure data services (Apache Spark, Azure Data Factory, Synapse Analytics, RDBMS such as SQL Server) Proven leadership and management experience in data engineering teams Proficiency in PySpark, Python (with Pandas), T-SQL … Ability to manage multiple projects and meet deadlines Azure certifications such as Microsoft Certified: Azure Data Engineer Associate or Azure Solutions Architect Nice to Have Experience with Scala for Apache Spark Knowledge of other cloud platforms like AWS or GCP Our Benefits Include Group pension plan, life assurance, income protection, and critical illness cover Private medical insurance and More ❯

work for the biggest multinational companies where years-long behemoth projects are the norm, our projects are fast-paced, typically 2 to 4 months long. Most are delivered using Apache Spark/Databricks on AWS/Azure and require you to directly manage the customer relationship alone or in collaboration with a Project Manager. Additionally, at this seniority … no shortage of learning opportunities at DATAPAO, meaning that you'll get access to Databricks' public and internal courses to learn all the tricks of Distributed Data Processing, MLOps, Apache Spark, Databricks, and Cloud Migration from the best. Additionally, we'll pay for various data & cloud certifications, you'll get dedicated time for learning during work hours, and … year , depending on your assessed seniority level during the selection process. About DATAPAO At DATAPAO, we are delivery partners and the preferred training provider for Databricks, the creators of Apache Spark. Additionally, we are Microsoft Gold Partners in delivering cloud migration and data architecture on Azure. Our delivery partnerships enable us to work in a wide range of industries More ❯

Kubernetes , and serverless architectures. Distributed Systems : Strong understanding of distributed systems, microservices architectures, and the challenges of building high-throughput, low-latency systems. Hands-on experience with tools like Apache Kafka , RabbitMQ , Apache Pulsar , and other messaging systems for real-time data streaming. DevOps and Infrastructure Automation : Expertise in DevOps principles, infrastructure-as-code, and automation tools such … Kubernetes . Experience with building, maintaining, and optimizing CI/CD pipelines. Big Data & Data Engineering : Strong background in processing large datasets and building data pipelines using platforms like Apache Spark , Databricks , Apache Flink , or similar big data tools. Experience with batch and stream processing. Security : In-depth knowledge of security practices in cloud environments, including identity More ❯

exciting and critical challenges to the UK’s digital landscape. This role requires strong expertise in building and managing data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. The successful candidate will design, implement, and maintain scalable, secure data solutions, ensuring compliance with strict security standards and regulations. This is a UK based onsite role with … the option of compressed hours. The role will include: Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. Implement data ingestion, transformation, and integration processes, ensuring data quality and security. Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards. Manage and … experience working as a Data Engineer in secure or regulated environments. Expertise in the Elastic Stack (Elasticsearch, Logstash, Kibana) for data ingestion, transformation, indexing, and visualization. Strong experience with Apache NiFi for building and managing complex data flows and integration processes. Knowledge of security practices for handling sensitive data, including encryption, anonymization, and access control. Familiarity with data governance More ❯

exciting, and critical challenges to the UK’s digital landscape. This role requires strong expertise in building and managing data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. The successful candidate will design, implement, and maintain scalable, secure data solutions, ensuring compliance with strict security standards and regulations. This is a UK-based onsite role with … the option of compressed hours. The role will include: Design, develop, and maintain secure and scalable data pipelines using the Elastic Stack (Elasticsearch, Logstash, Kibana) and Apache NiFi. Implement data ingestion, transformation, and integration processes, ensuring data quality and security. Collaborate with data architects and security teams to ensure compliance with security policies and data governance standards. Manage and … experience working as a Data Engineer in secure or regulated environments Expertise in the Elastic Stack (Elasticsearch, Logstash, Kibana) for data ingestion, transformation, indexing, and visualization Strong experience with Apache NiFi for building and managing complex data flows and integration processes Knowledge of security practices for handling sensitive data, including encryption, anonymization, and access control Familiarity with data governance More ❯

and solutions to improve our data infrastructure and capabilities. What is needed to succeed: Technical skills: Problem-solving team player with an analytical mind. Strong knowledge of SQL and Spark SQL. Understanding of dimensional data modelling concepts. Experience with Azure Synapse Analytics. Understanding of streaming data ingestion processes. Ability to develop/manage Apache Spark data processing More ❯

the UK for the last 10 years, and ability to obtain security clearance. Preferred Skills Experience with cloud platforms (IBM Cloud, AWS, Azure). Knowledge of big data frameworks (Apache Spark, Hadoop). Experience with data warehousing tools like IBM Cognos or Tableau. Certifications in relevant technologies are a plus. Additional Details Seniority level: Mid-Senior level Employment More ❯

working with relational and non-relational databases to build data solutions, such as SQL Server/Oracle , experience with relational and dimensional data structures. Experience in using distributed frameworks ( Spark, Flink, Beam, Hadoop ). Proficiency in infrastructure as code (IaC) using Terraform . Experience with CI/CD pipelines and related tools/frameworks. Containerisation Good knowledge of containers … AWS, or Azure . Good understanding of cloud storage, networking, and resource provisioning. It would be great if you had... Certification in GCP “Professional Data Engineer”. Certification in Apache Kafka (CCDAK). Proficiency across the data lifecycle. WORKING FOR US Our focus is to ensure we are inclusive every day, building an organisation that reflects modern society and More ❯

Azure cloud services across a mixture of Enterprise and SME environments Proficiency in Python, SQL, Azure Data Factory, Azure Synapse Analytics, Azure Data Lakes, and big data technologies like Apache Spark Experience with DevOps practices and CI/CD pipelines in an Azure environment is a plus. Certification in Azure (e.g., Microsoft Certified: Azure Data Engineer Associate) is More ❯

We implement the systems that require the highest data throughput in Java. Within Data Engineering we use Dataiku, Snowflake, Prometheus, and ArcticDB heavily. We use Kafka for data pipelines, Apache Beam for ETL, Bitbucket for source control, Jenkins for continuous integration, Grafana + Prometheus for metrics collection, ELK for log shipping and monitoring, Docker for containerisation, Kubernetes for container … of one or more relevant database technologies e.g. MongoDB, PostgreSQL, Snowflake, Oracle Proficient with a range of open source frameworks and development tools e.g. NumPy/SciPy/Pandas, Spark, Jupyter Advantageous Prior experience of working with financial market data or alternative data Relevant mathematical knowledge e.g. statistics, time-series analysis Experience in data visualisation and building web apps More ❯

SQL with atleast 5 years of experience • Working experience in Palantir Foundry platform is must • Experience designing and implementing data analytics solutions on enterprise data platforms and distributed computing (Spark/Hive/Hadoop preferred). • Proven track record of understanding and transforming customer requirements into a best-fit design and architecture. • Demonstrated experience in end-to-end data … management, data modelling, and data transformation for analytical use cases. • Proficient in SQL (Spark SQL preferred). • Experience with JavaScript/HTML/CSS a plus. Experience working in a Cloud environment such as Azure or AWS is a plus. • Experience with Scrum/Agile development methodologies. • At least 7 years of experience working with large scale software systems. More ❯