1 to 25 of 65 Remote/Hybrid Apache Jobs in the UK excluding London

replay, backfill, recoverability and schema evolution Define and enhance operational telemetry, monitoring and platform observability Streaming & Distributed Data Processing Build and optimise solutions using:Apache Kafka Apache NiFi Apache Spark/PySpark Hadoop ecosystem technologies Hive and large-scale data storage platforms Develop scalable matching, enrichment … interested in engineers who can demonstrate strong expertise across several of the following areas: Core Technical Skills Python development in enterprise environments Apache Spark/PySpark Apache Kafka Apache NiFi Hadoop and Hive Advanced SQL REST API integration and development Data modelling and data lineage Distributed systems ...

solving and communication skills. Able to work independently and remotely, and to travel to Newcastle once a week. Desirable skills and experience Experience with Apache Spark, Databricks or Kafka. Familiarity with orchestration tools such as Airflow. Experience with CI/CD for data pipelines and Infrastructure as Code. Knowledge ...

vector databases Streaming technologies (Kafka, Kinesis, Flink, etc.) Containers and related services (Docker, Kubernetes, container Registries, etc) Data modelling approaches (Kimball, Imnon) Orchestration tools - Apache Airflow, Prefect or cloud-native tools Backend software development (Java, APIs, Scalability, Logging and Monitoring etc.) MLFlow and other MLOps/Machine Learning Engineering ...

will be an integral part of our team dedicated to building scalable and secure data platforms. You will leverage your expertise in Databricks, Apache Spark, and Azure to design, develop, and implement data warehouses, data lakehouses, and AI/ML models that fuel our data-driven operations.We are looking ...

production pipelines Strong SQL and Python, plus hands-on Git Production experience with at least one of Databricks, Snowflake, Azure Data Factory, AWS Glue, Apache Airflow or PySpark Cloud-native orchestration and distributed data processing Infrastructure as code (e.g. Terraform) CI/CD and DevOps or DevSecOps practice ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like JSON ...

possess hands‐on experience with one or more of the following technological areas (or their equivalent cloud‐native alternatives): Analytics & Platform Frameworks like Apache Spark or similar cloud-specific services. Transformation & Orchestration tools Database Technologies Libraries & Schema Management Storage Formats: Familiarity managing diverse data types through formats like JSON ...

schemas for both structured and unstructured data with a snowflake environment Enhance cloud data capabilities, working with AWS S3, Azure Data Lake, and Apache Spark Collaborate with cross-functional teams to ensure data availability for advanced analytics and reporting Automate workflows, implement CI/CD pipelines, and maintain data ...

management and international regulatory compliance. YOUR SKILLS AND EXPERIENCE ESSENTIAL CRITERIA Strong experience designing, building and maintaining scalable data pipelines and integration solutions using Apache Airflow (or similar orchestration platforms), Python, SQL and bash/linux. Experience developing cloud-native data platforms and solutions on AWS, including working with ...

United Kingdom A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical and engineering skills to deliver trusted, well … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

engineering or business intelligence solutions. Strong proven commercial experience with Databricks. Strong experience developing ETL/ELT pipelines and data integration solutions. Experience using Apache Spark, PySpark, and Spark SQL. Strong SQL development and optimisation skills. Experience working with large and complex datasets from multiple source systems. Knowledge ...

with monitoring/logging tools (ELK stack, Prometheus, Splunk) Familiarity with service mesh and modern networking (Istio, Envoy) Exposure to data engineering and integration (Apache ecosystem, SQL, Python) Recognised technical certifications (AWS/Azure Architect, Kubernetes CKA/CKAD, Terraform) Experience in influencing technical standards, best practices, or contributing ...

creative mind. It would be nice if you had: It would be nice if you had: Data Engineering – experience of one or more: Apache ecosystem, SQL, Python. Web - HTML, CSS, JavaScript, XML, SOAP. Experience with Secure DevSecOps within an Agile/SAFe environment. Containerisation & Orchestration - Docker, Podman, Kubernetes, Rancher ...

Azure Cosmos DB, including SQL API and Mongo API. Excellent knowledge of Cosmos DB Change Data Feed (CDF) and Spark integration. Strong PySpark and Apache Spark distributed computing experience. Experience with Cosmos DB SDKs, including .NET, Java, Python or Node.js. Expertise in NoSQL data modelling, partitioning, indexing and query ...

equivalent apprenticeship/work experience in a subject such as Mathematics, Statistics, Computer Science, or Geography DESIRABLE Exposure to Big Data Analytics (BigQuery, Apache Spark), Parallel Computing (Kubernetes, Databricks), Cloud Engineering (AWS, GCP, Azure), Spatial Query Optimisation, Graph Computing, or Microservices Relevant industry sector knowledge across retail, property ...

Analytics, Artificial Intelligence or a related field, or equivalent experience, would be effective, although practical experience is equally valuable. “Nice to Have” Azure, Databricks, Apache Spark or comparable cloud and data platforms. Data pipelines, ETL and workflow‐orchestration technologies. Machine‐learning methods such as forecasting, prediction or classification. Experience ...

experience with BigQuery . Experience building and supporting production data pipelines. Experience developing transformation layers using dbt . Experience with orchestration tools such as Apache Airflow or Cloud Composer . Strong SQL and Python development skills. Experience designing scalable data models and data warehouse solutions. Strong stakeholder engagement ...

help design and deliver cutting-edge data solutions for customers operating in some of the UK's most complex and challenging environments. Using Databricks, Apache Spark and cloud-native technologies, you’ll develop scalable Lakehouse architectures and production-grade data pipelines that enable organisations to trust, share and exploit … opportunity to grow your career while helping customers unlock the power of their data. Responsibilities Design, build and optimise scalable data pipelines using Databricks, Apache Spark, PySpark and Delta Lake. Develop and implement modern Lakehouse architectures, applying Medallion (Bronze, Silver, Gold) design principles to support trusted and reusable data ...

colleagues Must-have skills: RabbitMQ Java and microservices MongoDB and MySQL Kubernetes Grafana and Prometheus GitLab Job scheduling tooling (eg JS7 Job Scheduler, TWS, Apache Airflow) Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects) Nice to have: Databases and data platforms, ideally Oracle or Cloudera ...

have skills: RabbitMQ Java/Microservices MongoDB/MySQL Kubernetes Grafana/Prometheus Gitlab Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). Nice to have: Experience with databases/data platforms ...

analytical and operational domains.Establish ELT/ETL and streaming architectures on Snowflake Snowpipe, Snowpipe Streaming, Dynamic Tables, Streams and Tasks, external tables, and Apache Iceberg tables along with orchestration standards (dbt, Airflow, Snowflake Tasks) and CI/CD/DataOps best practices.Implement security-by-design in Snowflake: RBAC … Azure, or GCP) including Snowflake deployment, networking (Private Link), and cloud storage integration (S3, ADLS, GCS); familiarity with lakehouse stacks (Databricks, Delta Lake, Apache Iceberg) is a plus.Mastery of Snowflake security and governance: RBAC, tag-based governance, dynamic data masking, row access policies, column-level security, Tri-Secret encryption ...

should have specialist experience in one or more of the following technologies**Azure Databricks*** Design and build high-performance data pipelines: Utilize Databricks and Apache Spark to extract, transform, and load data into Azure Data Lake Storage and other Azure services.* Experience of Databricks ML and Azure … develop predictive models and drive business insights.* Proven expertise in Databricks, Apache Spark, and data pipeline development and strong understanding of data warehousing concepts and practices.* Experience with Microsoft Azure cloud platform, including Azure Data Lake Storage, Databricks and Azure Data Factory.* Azure Data Engineer Associate and Databricks Certified ...

configuration management and an interest in migrating to Ansible and cloud-native tooling. Experience managing Kubernetes platforms and deploying containerised applications. Knowledge of GitLab, Apache NiFi, HashiCorp Vault, Trend Micro Deep Security Manager, Ivanti device control, and Ansible Tower/AWX. Windows Infrastructure Experience with Windows Server administration, deployment ...

support comprehensive data analysis and reporting. Implement and uphold best practices in data governance, security, and compliance, using tools like Azure Purview, UnityCatalog, and Apache Atlas tomaintaindata integrity and trust. Ensure data quality and integrity through meticulous attention to detail and rigorous QA processes, continually refining and optimising data … latency, such as managing 1 billion+ records or terabyte-sized databases. Knowledge of Delta TablesorApache Iceberg:Understanding and experience with Delta Tables or Apache Iceberg for managing large-scale data lakes efficiently. Knowledge of Data Governance Tools:Experience with data governance tools like Azure Purview, UnityCatalog, or Apache ...

model: Hybrid 3 days on site each week Rate: up to £530 per day --- Responsibilities Design, implement, and maintain high-throughput data pipelines using Apache Spark and cloud-native tools Develop ETL/ELT workflows for managing structured and unstructured data in AWS cloud environments Collaborate with AI/… analytics enablement --- Requirements (Must Have Skills) 5+ years of data engineering experience building high-performing and large-scale data solutions Hands-on expertise in Apache Spark development using Java or Scala (Python experience considered) Strong proficiency in AWS cloud services (S3, EMR, Glue, IAM, etc.) and cloud-native architecture ...