26 to 50 of 72 Apache Spark Jobs in the UK excluding London

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best ...

data engineering, ETL development or data platform delivery. Strong programming skills in Java or Python. Exposure to modern data technologies such as Kafka, Spark or Flink. Experience working within cloud environments, ideally AWS. Understanding of software engineering principles, version control and CI/CD practices. Familiarity with container technologies ...

evolving platform stack — from data ingestion and event processing to REST services and internal tooling. Build and maintain streaming and batch processing systems (Flink, Spark, Kafka) that power real‐time and analytical workloads. Take responsibility for all aspects of software engineering: design, implementation, QA, and maintenance, with … Willingness and ability to take on new technologies as our stack evolves. Nice to Have Experience with stream processing frameworks such as Flink or Spark Streaming. Experience with task orchestration tools such as Apache Airflow or similar systems. Experience working with and designing infrastructures for large‐scale data ...

teams to drive outcomes Core skills (indicative) Production pipeline design and deployment experience Strong engineering capability with Python, SQL, plus big data tooling (e.g., Spark, and Java/Scala where relevant AWS, Azure, GCP Working pattern Hybrid working, with the team on client site or in the office ...

transformation programmes. Strong stakeholder management and communication skills. Experience working across cross-functional delivery teams. Data Engineering & Platforms Databricks Delta Lake Unity Catalog dbt Spark/PySpark Data pipeline development Lakehouse and modern data platform architectures Cloud Technologies AWS S3 Glue Airflow Data Modelling & Architecture Data Vault Kimball Data ...

Excellent verbal and written communication skills BS or MS degree in Computer Science or equivalent Experience in distributed computing frameworks like Hive/Hadoop, Apache Spark Experience in developing Finance or HR related applications Experience with cloud services: AWS Elastic Beanstalk, EC2, S3, CloudFront, RDS, DynamoDB, VPC, Elastic … Cache, Lambda Working experience with Terraform Experience in creating workflows for Apache Airflow What the job involves As a senior technical lead, you consider yourself a platform services developer and develop innovative solutions using modern software frameworks and paradigms You are self‐driven and a problem solver, with ...

platforms that enable real‐time and batch data processing at scale. Responsibilities Drive platform capability improvements – continuously enhance distributed data platforms using knowledge of Spark, Databricks, Kafka, Pulsar, Cassandra, and object stores to maintain best‐in‐class capabilities. Enable application data consumption – understand how applications produce and consume data … structures, partitioning strategies, consistency models, and performance optimisation, as well as OLTP databases such as PostgreSQL and MySQL. Distributed compute frameworks – strong experience with Apache Spark and distributed data processing, including optimisation, performance tuning, and large‐scale batch/streaming workloads. Cloud‐native data platforms – demonstrated experience with ...

Security Groups, CloudFormation, RDS, DynamoDB and Redshift, including secure and well-governed implementations. Data engineering and pipelines, building efficient batch and streaming solutions using Apache Spark and AWS services, and working with large-scale datasets in enterprise environments. Demonstrated technical problem-solving capability, with the ability to rapidly ...

Scala. Strong knowledge of SQL and experience with data warehousing solutions. Familiarity with cloud platforms (AWS, Azure, Google Cloud) and big data technologies (Hadoop, Spark). Excellent problem-solving skills and a keen eye for detail. A collaborative spirit and a genuine passion for data! Our Client's Benefits ...

services, capital markets or FinTech. Additional AWS services such as SQS, SNS, Step Functions, EventBridge or Secrets Manager. Big data technologies including DuckDB, pandas, Spark, Databricks or Snowflake. AI/ML technologies including vector databases, semantic search or LLM orchestration frameworks. Continuous delivery and modern development workflows. Data science ...

machine learning development (training and/or deploying models at scale). Outstanding programming skills in Python. Experience with distributed data processing frameworks (e.g., Spark). Experience with deep learning/NLP frameworks (e.g., PyTorch, Hugging Face). Solid understanding of software engineering principles and ML lifecycle practices (MLOps ...

Essential Skills of the Analytical Platform Engineer Python Kubernetes Docker CI/CD pipelines (GitLab or similar) Desirable skills MLOps and data engineering concepts Apache Spark Scala S3-compatible object storage Graph database technologies If you are an driven and experienced Analytical platform engineer looking for their next ...

Essential Skills of the Analytical Platform Engineer Python Kubernetes Docker CI/CD pipelines (GitLab or similar) Desirable skills MLOps and data engineering concepts Apache Spark Scala S3-compatible object storage Graph database technologies If you are an driven and experienced Analytical platform engineer looking for their next ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

unstructured workloads at scale - Integrate multiple sources into modern cloud platforms - Collaborate across engineering and delivery on mission-critical programmes **The environment** Python, SQL, Spark, Java or Scala · AWS, Azure or GCP · modern distributed and cloud-native architectures **What we need** - Active DV clearance essential - Production-scale pipeline experience ...

Strong commercial experience in data engineering and modern data platforms. Advanced programming skills in Java or Python. Expertise working with technologies such as Kafka, Spark, Flink or similar distributed data frameworks. Experience delivering solutions within AWS or other major cloud platforms. Strong understanding of data architecture, scalability and performance ...

experience applying them with packages like Pandas, sci-kit-learn, SciPy) is highly beneficial Experience processing big data, ideally in a Hadoop/Spark environment, would be beneficial Proven experience implementing CI/CD and DevOps practices, with the ability to improve deployment reliability, testing standards, and release processes ...

Candidate Requirements: Extensive experience building and deploying machine learning models in production, with strong hands-on Python capability. Experience with distributed computing frameworks (e.g., Spark or similar) and scalable data processing environments. Exposure to cloud-based platforms (e.g., AWS, Azure, GCP or Databricks) and modern ML/MLOps practices. ...

platforms (AWS, Azure, or GCP). Distributed Systems Mastery: Deep technical capability in structuring data platforms, data pipelines, and high‐throughput distributed systems (like Spark or Kafka) optimized for performance and scalability. Cross‐Functional Technical Leadership: Exceptional ability to bridge the gap between deep technical engineering and strategic business ...

writing, talks, or threads about building with AI. MLOps and model serving experience (BentoML, MLflow, Vertex AI, SageMaker). Streaming and batch ingestion pipelines (Spark, Airflow, Beam, Glue). Healthcare or life sciences domain exposure. AWS Professional certification or other relevant industry certifications. What We Offer At Newpage ...

Codebase ownership Technical Application ownership Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins, Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

Python. Experience with Kubernetes and Docker. Understanding of CI/CD pipelines (e.g. GitLab). Exposure to data platforms, MLOps or machine learning environments. Spark or Scala. AWS services (e.g. S3). Elasticsearch or graph databases. OIDC/OAuth. Node.js or React. To apply for this Analytics Platform Engineer ...

help deliver robust, secure and high-performing data products. What youll be doing: Design and build scalable data pipelines using Python, SQL, Spark and Databricks. Develop, deploy and optimise machine learning models, including NLP and generative AI solutions. Build data ingestion pipelines across batch and streaming environments. Design data … visualisations using Power BI or Tableau. Mentor junior team members and promote engineering best practices. What you need: Commercial experience with Python, SQL, Spark and modern data engineering frameworks. Experience deploying machine learning models into production. Strong knowledge of cloud platforms such as Azure, AWS or GCP. Experience with ...

Months contract 2 Days a month onsite - Expenses to Newcastle depends on their hub. Start 3 from offer Experience required AWS Services Terraform Apache Spark Python Airflow Docker Nice to haves Gitlab Security Scanning Agents (eg Trivy, Trend Micro, Wiz etc) Jupyter Notebooks Integration with DWP Integration Team ...

kubernetes) Experience with Key value stores Experience in git and basic Unix Commands You might also have (Desirable) Experience with large data processing technologies (Apache Spark) Other helpful information Hybrid Working: At EMBL-EBI we are pleased to offer hybrid working options for all our employees. A dedicated ...