1 to 25 of 125 Apache Jobs in England

detect threats, analyze patterns, and respond to incidents in real-time. You'll be designing and developing highly scalable streaming services using technologies like Apache Kafka, Apache Flink, Apache Storm and Apache Spark in Java, Python and Go. You'll design and implement data pipelines that … Design, develop, and maintain streaming microservices in Java and Go that provide real-time data processing capabilities to internal teams Build and optimize Apache Flink and Apache Spark based data pipelines running on Kubernetes infrastructure Design event-driven systems leveraging Kafka for high-throughput, low-latency message streaming ...

Apache Spark Python AWS Cloud Data Pipelines A hands-on data engineering role within a large-scale cloud data programme, responsible for building, maintaining, and troubleshooting data pipelines using Apache Spark, PySpark, Apache Airflow, and a broad suite of AWS services. You will apply strong analytical … complex, cloud-based data programme - designing, building, and maintaining data pipelines that process large volumes of data across a modern AWS-native stack. Using Apache Spark and PySpark for distributed data processing, Apache Airflow for orchestration, and a range of AWS services for storage, compute, and analytics ...

understanding of Quantexa components such as: Entity Resolution (ER), Scoring Framework, Contextual Network Generation, UI Configuration. Strong background in Big Data technologies, including: Scala, Apache Spark, Hadoop, ElasticSearch. Experience with data modelling and design in the context of large-scale analytics platforms. Strong stakeholder management and communication skills – capable ...

market data engineering with proven experience building reliable, scalable data platforms. Our non-negotiable requirements are: Expert Python Strong SQL Scalable data pipeline development Apache Parquet Market data normalisation Automated data quality controls Deep tick and market data expertise Experience in MLOps is advantageous but not essential. We welcome ...

/CD pipelines and deployment automation. • Familiarity with monitoring, logging, and observability tools. • Awareness of application platforms and middleware such as Java, Tomcat, Apache, and NodeJS. • Experience using Git and collaborative development workflows. • Basic scripting skills using Shell or Python. • Understanding of containerisation concepts using Docker. • Strong problem-solving ...

Ansible for repeatable, automated provisioning of GPU clusters and AI platform environments; GitOps (ArgoCD) for continuous, declarative platform delivery. Pipeline orchestration: Kubeflow Pipelines, Apache Airflow, or Argo Workflows to orchestrate multi-stage training, fine-tuning, and inference pipelines. Cluster & workload scheduling: Slurm, Run:ai, and NVIDIA Base Command Manager ...

developing others. Desirable to have skills or experience in: - SAS Viya for data analytics. - Job Scheduling tooling (e.g. Berlin/JS7 Job Scheduler, TWS, Apache Airflow). - Denodo for data virtualisation. - Reporting and visualisation tooling (from Pentaho BA, Power BI, SAP Business Objects). - ALM Tooling (Jira, Confluence). ...

Python skills * Hands-on experience designing and developing ETL pipelines and data workflows * Experience designing and maintaining data pipelines using technologies such as Apache Airflow or Spark * Experience working with cloud platforms including AWS, Azure or GCP * Strong understanding of data modelling, integration and governance principles * Experience working with ...

ownership Technical Application ownership Qualifications Strong experience with DevOps & IaC tooling - Terraform, CI/CD pipelines, Git, Jenkins , Harness Strong experience of developing on Apache Spark/Hadoop Strong experience with AWS cloud services including AWS Glue, S3, Step Strong experience building data pipelines and data lake Experience ...

practices. Proven experience in designing, implementing, and maintaining Software solutions. Hands-on experience with any cloud, AWS is preferred. Advanced proficiency in Python and Apache Spark, with a strong focus on ETL data processing and scalable data engineering workflows. Our Package 10% on target bonus A BT Pension scheme ...

alerting. Minimum requirements: 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt; Working knowledge of Python, SQL and shell scripting; Experience with Docker/k8s and “infrastructure as a code ...

with the ability to deliver incremental data modelling solutions in collaboration with engineering teams. Technical Environment (Desirable) Microsoft Fabric Azure Data Factory Python PySpark Apache Spark Desirable Skills Azure DevOps Contract Details Sector: Banking/Financial Services Location: London – Office based 4 days per week Contract Type: TBC Start ...

PostgreSQL (or similar relational databases) and NoSQL databases. Experience with vector databases such as Qdrant, Milvus or pgvector . Familiarity with technologies such as Apache Spark, Airflow, Kafka, Elasticsearch/OpenSearch, Pandas and Polars . Strong understanding of data modelling, storage architecture and performance optimisation. Desirable Experience working ...

ability to help refine problem statements and propose solutions taking effort-impact-scalability tradeoff into account. Some skills that will make you stand out: Apache Spark, Iceberg, Kafka, dbt Scikit-Learn, XGBoost, PyTorch, MLFlow,, GraphFrames, Ray AWS (S3, EMR, SageMaker, Lakeformation), Terraform, Docker, GitHub CI/CD Knowledge Graphs ...

Proven track record with data governance, quality, lineage, and security practices. Experience with real-time/streaming data technologies, data Ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub). Strong proficiency in relational and non-relational databases (e.g., PostgreSQL, MongoDB, Cassandra). Deep knowledge ...

reviews and continuous improvement Skills & Experience * Commercial experience in Data Engineering * Strong Python or Java development skills * Experience with AWS cloud services * Knowledge of Apache Spark, Hadoop, NiFi or similar big data technologies * Experience building ETL/ELT pipelines and processing large datasets * Understanding of software engineering best practices ...

innovation-led working environment Key Experience Proven experience as a Data Architect, Databricks Architect, Lead Data Engineer or Solutions Architect Strong expertise in Databricks, Apache Spark, PySpark and SQL Experience designing and implementing modern data lake or lakehouse architectures Hands-on experience with Kafka Strong understanding of AWS data ...

Ideally will be a bonus if you have the following... Experience or knowledge of FluxCD, GitOps workflow, Appgate, MongoDB, Airflow (MWAA), Snowflake Experience in Apache Airflow (MWAA), AWS Backup and S3 versioning for business continuity. Programming experience in C#, Python and/or SQL Experience in Keycloak or other ...

data transformation tools such as Spark and DBT, with a focus on building well-tested, maintainable data products is a strong advantage. Familiarity with Apache Iceberg table format and schema management via catalogs such as AWS Glue, with an appreciation for data quality and consistency is a strong advantage. ...

Proven track record with data governance, quality, lineage, and security practices. Experience with real-time/streaming data technologies, data Ingestion/ETL (e.g., Apache Kafka, Apache NiFi, Kinesis, Pub/Sub). Experience in, or knowledge of, DevSecOps Tooling and Processes. Self-starter with the ability ...

Understanding of Azure data services and API-based integration. Next.js 14, FastAPI Python, Pipeline orchestration, Data transformation, Data quality, Casbin RBAC, Kuzu knowledge graph, Apache Iceberg, MCP AI server integration and ODCS v2 data contracts layer. Experience in establishing the infrastructure, pipelines, processes, and governance required for production environment ...

software and site reliability engineering. Preferred experience with distributed ML frameworks such as Horovod or TensorFlow Distributed, familiarity with data engineering pipelines such as Apache Airflow or Spark, and knowledge of chaos engineering tools and compliance frameworks such as GDPR, SOC 2, or ISO 27001. Benefits Relocation benefits available ...

with data scientists, analysts, and business stakeholders. Strong understanding of data platform reliability, performance, scalability, and cost optimisation. Experience with technologies such as ClickHouse, Apache Iceberg, BigQuery, dbt, Elasticsearch, Druid, GCP, or AWS is highly desirable. If you're an experienced Senior Data Engineer looking to lead complex data ...

production, providing subject matter expertise on the .Net stack and contributing to technical design discussions. You'll use a range of technology including Apache Flink with Java for large scale data processing and will be able to assess and recommend new and emerging technologies, using the best tool ...

best practices and coding standards. Interface with other technology teams to extract, transform, and load data from a wide variety of data sources using Apache Airflow, Spark, SQL, Python, and ETL tools. CI/CD & DevOps Design and implement CI/CD pipelines for data platforms. Automate code deployment ...