1 to 25 of 100 Apache Spark Jobs in the UK

develop, and maintain end-to-end data pipelines using Ab Initio (GDE, PDL, Conduct>IT, Express>IT).Build scalable batch and streaming workloads using Apache Spark (PySpark/Scala).Develop and orchestrate data ingestion flows using Apache NiFi for real-time and near-real-time use cases.Optimize …/ELT jobs for scalability, throughput, and resource efficiency.Perform bottleneck analysis and tuning for Spark jobs, NiFi processors, and Ab Initio graphs.Demos and knowledge transfers to stakeholders around product developments.Spread your experience and knowledge to junior engineers.Improve automated testing and deployments using CI/CD pipelines.Expectation to line manage ...

intersection of consulting, technology, innovation, and strategic alliances and you will bring deep expertise in the Databricks Data Intelligence Platform, Lakehouse architecture, Delta Lake, Apache Spark, Unity Catalog, MLflow, Mosaic AI, data engineering, machine learning, GenAI and cloud-native delivery.As a key leader within our AI & Data Practice … translate business needs into Databricks-led solution designs, build prototypes and proof points.• Shape end-to-end solutions using Databricks Intelligence Platform, Delta Lake, Apache Spark, Unity Catalog, Databricks Workflows, Delta Live Tables/Lakeflow-style pipelines, MLflow, Feature Store, Model Serving and Mosaic AI capabilities.Team Leadership & Delivery ...

Databricks-based Lakehouse that drives real-time data, analytics, and reporting across the Tote.You’ll build streaming and batch data pipelines on AWS using Apache Spark Structured Streaming, Delta Live Tables (DLT), and Kafka (MSK), ensuring our business teams across Liquidity, Product, Marketing, Finance, and Compliance have fast … deliver scalable, secure, and governed data solutions under Unity Catalogue.What you’ll be doing:You’ll design, build, and optimise data pipelines using Databricks, Spark Structured Streaming, and Delta Live Tables to ensure data flows efficiently and reliably across the organisation. By applying the Medallion Architecture, you’ll develop ...

applications, working closely with data engineers, analysts, and platform teams. Day to day duties Developing and maintaining data pipelines and processing jobs using Apache Spark and Python Orchestrating workflows with Apache Airflow Provisioning and managing cloud infrastructure on AWS using Terraform (Infrastructure as Code) Containerising applications … teams in an Agile environment, contributing to design reviews, code reviews, and documentation Essential Skills & Experience Strong hands-on experience with Python Experience with Apache Spark for large-scale data processing Practical knowledge of AWS services (eg S3, Lambda, EC2, Glue, RDS - adjust as relevant) Experience writing ...

databases utilised in modern AI applications. Strong Python development skills. Data & Infrastructure: Experience with some of the following technologies is highly desirable: Frameworks & Infrastructure: Apache Spark Apache Airflow Kafka Elasticsearch/OpenSearch Databases: PostgreSQL MongoDB Vector databases such as Qdrant, Milvus, or pgvector Python Ecosystem: Pandas Polars ...

data warehouses and data lakes. (Preferred)Experienced in data ingestion, processing, transformation, and delivery using batch, streaming, and real-time methods. (Preferred)Experience supporting Spark‐based platforms (e.g. EMR, Glue, Databricks) (Preferred)Proficient with DevOps tools and practices (Git, Jenkins, Docker, Kubernetes) as well as ETL frameworks (Apache Spark, Airflow, AWS IoT Analytics/Device Management/Events). (Preferred)Programming expertise in Python, Java, Scala, SQL, Spark, Unix shell scripting, and AWS CLI. (Preferred)Great background in building and managing AWS platforms for analytics and data lake workloads, including application and framework management. (Preferred ...

Testing Guild with common standards such as metrics and use of test tools Here's a snapshot of the technologies that we use: Scala Apache Spark Databricks Apache Parquet Azure Cloud Platform Azure DevOps (Test plans, Backlogs, Pipelines) GIT We'd love to hear from ...

semi-structured, and unstructured dataHands-on expertise in containerization and orchestration using Docker and Kubernetes for scalable production systemsFamiliarity with distributed computing frameworks (e.g., Spark, Dask), cloud platforms (e.g., AWS, Azure, GCP), and analytics libraries (e.g., pandas, numpy, matplotlib)Exposure to DevOps, DataOps, and MLOps concepts and best practicesExperience ...

standards. Skills Strong experience in SQL and data modeling concepts. Hands-on experience with ETL/ELT development. Proficiency in Python, Scala, or Spark-based data processing. Experience with cloud platforms such as AWS, Azure, or GCP. Knowledge of Data Warehouse and Data Lake architectures. Experience with orchestration tools ...

/ELT processes and data pipeline development Experience working with cloud platforms such as AWS, Azure or GCP Knowledge of technologies such as Spark, Kafka, Airflow or similar Experience with databases, data modelling and data warehouse solutions Understanding of CI/CD, software engineering best practices and secure development ...

able to guide how one could deploy infrastructure into different environments. Knowledge of handling and transforming various data types (JSON, CSV, etc) with Apache Spark, Databricks or Hadoop. Good understanding of possible architectures involved in modern data system design (Data Warehouse, Data Lakes, Data Meshes) Ability to create ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills:- Proficient in Python and SQL, with experience in Apache Spark.- Strong experience building backend services and APIs. Database Expertise:- Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

quality and consistency across platforms while complying with governance policies. What You Bring: Core Development Skills: - Proficient in Python and SQL, with experience in Apache Spark. - Strong experience building backend services and APIs. Database Expertise: - Solid understanding of relational databases (SQL Server, PostgreSQL) and distributed NoSQL databases (Cassandra, MongoDB ...

Azure) Experience with: o Data warehousing and lake architectures o ETL/ELT pipeline development o SQL and NoSQL databases o Distributed computing frameworks (Spark, Kinesis etc) o Software development best practices including CI/CD, TDD and version control. o Containerisation tools like Docker or Kubernetes o Experience ...

engineering best practices Skills & Experience * 3+ years' commercial experience in Data Engineering * Strong Java or Python development skills * Experience with Kafka, Flink or Spark * Experience building ETL/ELT or streaming data pipelines * Knowledge of AWS (Azure or GCP also considered) * Experience with Docker, Kubernetes and CI/ ...

Have: Certifications: Google Cloud or AWS Certified Solutions Architect certification. ETL and Data Pipeline Expertise: Experience with ETL tools and Hadoop-based technologies (e.g., Spark) and/or batch/streaming data pipelines such as Beam and Flink. Data Architecture: Proven ability in designing and building data lake ...

reconciliation and root-cause analysis.Experience building or supporting production data pipelines in a collaborative engineering environment.Experience working with distributed data processing frameworks such as Apache Spark.Working knowledge of common data formats such as JSON, Avro and Parquet.Technology EnvironmentThe role will involve working with a modern and evolving data stack. … should bring relevant experience and the ability to work across comparable technologies.Examples of technologies in scope include:Data processing and logic: ANSI SQL, Apache Spark, KafkaData formats: JSON, Avro, ParquetPlatforms and storage: Snowflake, Apache Iceberg, Databricks, Hadoop ecosystem technologies, Sybase IQEngineering and deployment: CI/CD tooling ...

architecture (SQL and NoSQL) Experience with cloud technologies Experience with cloud platforms Experience with CI/CD practices Experience with big data tools e.g. Spark, Hadoop, NiFi Experience managing direct reports, providing regular guidance and performance feedback Ability to create delivery plans with estimated timelines for personal and team ...

volume, data-intensive environments. Extensive AWS experience including services such as Glue, Lambda, S3, Redshift, and wider cloud-native data tooling. Strong knowledge of Apache Spark and large-scale data processing. Experience working with streaming technologies such as Kafka. Expert SQL and data modelling capabilities. Experience designing modern ...

range of projects that include internal and external suppliers. Essential Skills Proven experience as a Data Engineer Experience with Azure Data Factory, Databricks, or Apache Spark, following modern ETL/ELT principles. Experience with Microsoft Fabric Experience of using programming languages such as Python, Scala and SQL. Demonstrable ...

What we’re looking for: Strong Python development and software engineering experience Experience building scalable data pipelines and distributed data platforms Knowledge of Kafka, Spark and modern data processing technologies Experience with orchestration tools such as Airflow, Dagster or Prefect Hands-on experience with Docker, Kubernetes and containerised deployments ...

twin platforms. Experience with vector databases, embeddings, semantic search, or RAG. Experience with streaming data platforms such as Kafka, Pub/Sub, Flink, or Spark Streaming. Experience supporting closed-loop automation, predictive assurance, or self-healing network use cases. Experience working with OSS systems, inventory platforms, assurance systems ...

Experience with data warehousing concepts and technologies.Knowledge of other BI tools (e.g., Tableau, Looker) and data modelling techniques.Familiarity with big data technologies (e.g., Hadoop, Spark).Familiarity with modern data architecture (e.g., Data Mesh, Fabric).An understanding of infrastructure as code (e.g., Terraform)Experience with CI/CD pipelines ...

GCP. MSc or equivalent experience in Data Science, Statistics, Mathematics, Computer Science, Artificial Intelligence, Physics, Engineering or a related quantitative discipline. Experience with Spark or distributed data processing frameworks. Experience preparing data pipelines or analytical datasets to support machine learning workloads. Awareness of data engineering, data architecture, data governance ...

platform and lakehouse transformation journeys What We're Looking For Strong commercial experience in Data Engineering Hands-on experience with Databricks and/or Apache Spark Strong PySpark and SQL development skills Experience designing, building and supporting data pipelines Experience working with Azure or AWS Good understanding ...