[Paper Review] Topology-Aware Node Selection for Data Regeneration in Heterogeneous Distributed Storage Systems

Qingyuan Gong, Jiaqi Wang+4 more|arXiv (Cornell University)|Jun 18, 2015

Advanced Data Storage Technologies2 references4 citations

TL;DR

This paper proposes topology-aware node selection algorithms—SPSN and FLEX—to minimize data regeneration time in heterogeneous distributed storage systems by strategically choosing providers and newcomers based on link bandwidth heterogeneity. By avoiding low-capacity links and dynamically adjusting data flow, the schemes reduce regeneration time by up to 62.5% in highly heterogeneous networks compared to random selection.

ABSTRACT

Distributed storage systems introduce redundancy to protect data from node failures. After a storage node fails, the lost data should be regenerated at a replacement storage node as soon as possible to maintain the same level of redundancy. Minimizing such a regeneration time is critical to the reliability of distributed storage systems. Existing work commits to reduce the regeneration time by either minimizing the regenerating traffic, or adjusting the regenerating traffic patterns, whereas nodes participating data regeneration are generally assumed to be given beforehand. However, such regeneration time also depends heavily on the selection of the participating nodes. Selecting different participating nodes actually involve different data links between the nodes. Real-world distributed storage systems usually exhibit heterogeneous link capacities. It is possible to further reduce the regeneration time via exploiting such link capacity differences and avoiding the link bottlenecks. In this paper, we consider the minimization of the regeneration time by selecting the participating nodes in heterogeneous networks. We analyze the regeneration time and propose node selection algorithms for overlay networks and real-world topologies. Considering that the flexible amount of data blocks from each provider may deeply influence the regeneration time, several techniques are designed to enhance our schemes in overlay networks. Experimental results show that our node selection schemes can significantly reduce the regeneration time for each topology, especially in practical networks with heterogeneous link capacities.

Motivation & Objective

• Address the limitation of existing regenerating code schemes that assume fixed provider and newcomer selection, ignoring the impact of link capacity heterogeneity on regeneration time.
• Reduce data regeneration time in distributed storage systems by optimizing node selection to avoid low-bandwidth links and minimize bottlenecks.
• Design adaptive node selection algorithms that account for varying link capacities and dynamically adjust data flow to improve performance in real-world data center topologies.
• Evaluate the proposed schemes in both overlay networks and real-world Fat-tree architectures to validate performance gains under realistic bandwidth heterogeneity.
• Demonstrate that intelligent node selection can significantly outperform random selection, especially in highly heterogeneous environments.

Proposed method

• Propose the SPSN (Selective Provider and Newcomer) algorithm to select providers and newcomers in overlay networks by prioritizing high-capacity links and avoiding bandwidth bottlenecks.
• Introduce the FLEX (Flexible End-to-end Traffic) algorithm that dynamically computes data block sizes based on available link capacities to further reduce regeneration time.
• Model the regeneration process as a flow optimization problem where the bottleneck link determines the overall regeneration time, and aim to maximize the minimum link capacity in the repair topology.
• Use a uniform distribution for link capacities in simulations, with values ranging from [10,120] Mbps, to evaluate performance under varying degrees of heterogeneity.
• Implement and evaluate the schemes in NS2-based simulations, comparing them against random selection and baseline schemes in both synthetic overlay and real Fat-tree topologies.
• Extend the approach to Fat-tree architectures by modeling three-tiered bandwidth heterogeneity (server-to-switch, switch-to-switch, and core links) with increasing bandwidth ratios of 1:5:10.

Experimental results

Research questions

• RQ1 How does the selection of providers and newcomers affect regeneration time in heterogeneous distributed storage systems with variable link capacities?
• RQ2 Can intelligent node selection that avoids low-capacity links significantly reduce regeneration time compared to random selection?
• RQ3 To what extent can dynamic data flow adjustment based on available link capacity further improve regeneration performance in heterogeneous networks?
• RQ4 How do network scale, number of providers (d), and coding parameters influence the effectiveness of topology-aware node selection?
• RQ5 How does the performance of the proposed schemes vary across different levels of bandwidth heterogeneity and in real-world data center topologies like Fat-tree?

Key findings

• The FLEX algorithm reduces regeneration time by up to 62.5% compared to random selection in highly heterogeneous networks with link capacities in [10,120] Mbps.
• In networks with moderate heterogeneity ([50,120] Mbps), the performance gap between FLEX and SPSN narrows, becoming negligible at [90,120] Mbps.
• The SPSN-F algorithm significantly reduces regeneration time in Fat-tree topologies, with improvements increasing in larger networks and higher values of d.
• Regeneration time decreases as the number of providers (d) increases, due to reduced data load per provider, and this trend is amplified by topology-aware selection.
• In Fat-tree simulations, SPSN-F reduces regeneration time by a factor of 2–3 compared to random selection, with the largest gains observed in highly heterogeneous bandwidth environments.
• The regeneration time is highly sensitive to node selection: random selection can lead to bottlenecks as low as 30 Mbps, resulting in 4 seconds of regeneration time, while optimal selection achieves 1.5 seconds.