6 of 6 Apache Spark Jobs in Bristol

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations. Design … build high-performance data platforms: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Design and oversee the delivery of secure data warehouses and data lakehouses: Implement data models, data quality checks, and governance practices to ensure ...

should have specialist experience in one or more of the following technologies Azure Databricks Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services. Experience of Databricks ML and Azure … develop predictive models and drive business insights. Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices. Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory. Azure Data Engineer Associate and Databricks ...

data processing solutions using BigQuery, Pub/Sub, Dataflow and Dataproc Experience developing end‐to‐end solutions using batch and streaming frameworks such as Apache Spark and Apache Beam. Expert understanding of when to use a range of data storage technologies including relational/non‐relational, document ...

version control. Data Engineering Stack Experience building pipelines using streaming frameworks, distributed processing engines and relational or analytical storage technologies. Examples include Kafka, Spark, PostgreSQL. Orchestration Experience defining and running data workflows using modern orchestration frameworks. Examples include Airflow, Dagster or Prefect. Data Quality and Lineage Familiarity with tools ...

data pipeline orchestration tools Containerisation (Docker, EKS, AKS, Kubernetes) Proficiency in CI/CD principles and tools Familiarity with open-source data tools (e.g., Spark, Kafka, PostgreSQL) Infrastructure as Code (Terraform, Ansible) Air-gapped development Automation Client/stakeholder management Public sector domain understanding Technical architecture Experience: You will ...

writing, talks, or threads about building with AI. MLOps and model serving experience (BentoML, MLflow, Vertex AI, SageMaker). Streaming and batch ingestion pipelines (Spark, Airflow, Beam, Glue). Healthcare or life sciences domain exposure. AWS Professional certification or other relevant industry certifications. What We Offer At Newpage ...