[Paper Review] Hydra -- A Federated Data Repository over NDN

Justin Presley, Xi Wang+9 more|arXiv (Cornell University)|Nov 2, 2022

Caching and Content Delivery4 citations

TL;DR

Hydra is a decentralized, federated data repository built on Named Data Networking (NDN) that enables secure, scalable, and resilient scientific data sharing across distributed communities. By leveraging NDN’s data-centric security, in-network caching, and the State Vector Sync (SVS) protocol, Hydra automates replication and failure recovery, achieving high performance in file insertion (up to 135.65 Mbps for 100 MB files) and fast caching-based retrieval (up to 185.82 Mbps).

ABSTRACT

Today's big data science communities manage their data publication and replication at the application layer. These communities utilize myriad mechanisms to publish, discover, and retrieve datasets - the result is an ecosystem of either centralized, or otherwise a collection of ad-hoc data repositories. Publishing datasets to centralized repositories can be process-intensive, and those repositories do not accept all datasets. The ad-hoc repositories are difficult to find and utilize due to differences in data names, metadata standards, and access methods. To address the problem of scientific data publication and storage, we have designed Hydra, a secure, distributed, and decentralized data repository made of a loose federation of storage servers (nodes) provided by user communities. Hydra runs over Named Data Networking (NDN) and utilizes the State Vector Sync (SVS) protocol that lets individual nodes maintain a "global view" of the system. Hydra provides a scalable and resilient data retrieval service, with data distribution scalability achieved via NDN's built-in data anycast and in-network caching and resiliency against individual server failures through automated failure detection and maintaining a specific degree of replication. Hydra utilizes "Favor", a locally calculated numerical value to decide which nodes will replicate a file. Finally, Hydra utilizes data-centric security for data publication and node authentication. Hydra uses a Network Operation Center (NOC) to bootstrap trust in Hydra nodes and data publishers. The NOC distributes user and node certificates and performs the proof-of-possession challenges. This technical report serves as the reference for Hydra. It outlines the design decisions, the rationale behind them, the functional modules, and the protocol specifications.

Motivation & Objective

• Address the inefficiencies and bottlenecks of centralized and ad-hoc data repositories used by scientific communities.
• Reduce the burden of data publication and management by enabling decentralized, community-driven data sharing.
• Improve data findability, reusability, and performance through data-centric naming and in-network caching.
• Ensure strong security and trust through data-centric security, cryptographic signing, and a Network Operation Center (NOC) for credential management.
• Achieve system resilience and scalability via automated replication and failure detection using the SVS protocol.

Proposed method

• Deploy a federated network of storage nodes managed by scientific communities, each contributing resources to the Hydra federation.
• Use Named Data Networking (NDN) as the underlying transport layer to enable content-based addressing, in-network caching, and data-centric security.
• Implement the State Vector Sync (SVS) protocol to maintain a consistent global view of the system state across all nodes.
• Apply the 'Favor' metric—a locally computed value based on node attributes—to guide intelligent, automated replication decisions.
• Secure the system via a Network Operation Center (NOC) that issues and validates node and user certificates through proof-of-possession challenges.
• Support data operations (query, insert, fetch, delete) using NDN-based commands, with data split into chunks and stored across multiple nodes.

Experimental results

Research questions

• RQ1 How can a decentralized, federated data repository be built to reduce reliance on centralized or ad-hoc data management systems in scientific communities?
• RQ2 To what extent can NDN’s in-network caching and data anycast improve data retrieval performance and scalability?
• RQ3 How effective is the 'Favor' metric in guiding efficient, resilient, and automated data replication across distributed nodes?
• RQ4 Can the SVS protocol maintain a consistent global system state with low latency and high availability in a dynamic, decentralized environment?
• RQ5 What is the performance overhead and reliability of Hydra’s secure, automated data publishing and replication pipeline in real-world testbed conditions?

Key findings

• Hydra achieved an average file insertion speed of 135.65 Mbps for 100 MB files on the primary replica (rep1), with 61.72 Mbps on the secondary replica (rep2), demonstrating strong write scalability.
• File retrieval performance improved significantly with caching: up to 185.82 Mbps for 100 MB files when fetched from cache, compared to 13.85 Mbps without caching.
• For large files (1 GB), caching provided minimal benefit due to limited cache size (500 MB), resulting in similar speeds (13.37 Mbps uncached vs. 13.51 Mbps cached).
• The system successfully maintained consistent state across nodes using the SVS protocol, enabling reliable failure detection and automatic replication.
• The Hydra client commands (query, insert, fetch, delete) were validated across multiple testbed topologies and data sizes, confirming functional correctness and robustness.
• The deployment on the FABRIC testbed demonstrated the feasibility of Hydra as a production-ready, secure, and scalable data sharing platform for scientific workloads.