[Paper Review] A realistic distributed storage system that minimizes data storage and repair bandwidth

Bernat Gastón, Jaume Pujol+1 more|arXiv (Cornell University)|Jan 8, 2013

Advanced Data Storage Technologies3 references4 citations

TL;DR

This paper proposes a two-rack distributed storage model that minimizes both data storage and repair bandwidth by introducing rack-aware repair costs—where intra-rack communication is cheaper than inter-rack. It generalizes the threshold function for regenerating codes to handle asymmetric repair costs, demonstrating that the new model reduces repair cost compared to prior basic and static-cost models, especially when $k > d_1$. The model achieves lower repair bandwidth and cost through optimized helper node selection based on rack location.

ABSTRACT

In a realistic distributed storage environment, storage nodes are usually placed in racks, a metallic support designed to accommodate electronic equipment. It is known that the communication (bandwidth) cost between nodes within a rack is much lower than the communication (bandwidth) cost between nodes within different racks. In this paper, a new model, where the storage nodes are placed in two racks, is proposed and analyzed. In this model, the storage nodes have different repair costs to repair a node depending on the rack where they are placed. A threshold function, which minimizes the amount of stored data per node and the bandwidth needed to regenerate a failed node, is shown. This threshold function generalizes the threshold function from previous distributed storage models. The tradeoff curve obtained from this threshold function is compared with the ones obtained from the previous models, and it is shown that this new model outperforms the previous ones in terms of repair cost.

Motivation & Objective

• To address the gap in realistic distributed storage models by incorporating rack-based communication cost differences.
• To model storage systems where intra-rack and inter-rack repair bandwidths differ significantly, reflecting real-world data center topologies.
• To generalize the threshold function for regenerating codes to handle variable repair costs based on rack location.
• To derive and analyze the tradeoff between stored data per node and repair bandwidth under rack-aware costs.
• To demonstrate that the proposed rack model reduces repair cost compared to previous models, especially when $k > d_1$.

Proposed method

• Proposes a two-rack model where storage nodes are partitioned into two groups with distinct repair costs: $C_c$ for intra-rack and $C_e$ for inter-rack communication.
• Introduces a generalized threshold function $f(i)$ that determines the minimum repair bandwidth $\gamma = d\beta$ for each $i = 0, \dots, k-1$, based on the number of helpers from each rack.
• Derives the effective repair bandwidth $\beta_e = f(i)$ and repair cost $C_T^1, C_T^2$ for each rack, using parameters $d_1, d_2, \tau = C_e/C_c$, and $L[i](k-i) + g(i)$ as a normalization factor.
• Applies min-cut analysis on information flow graphs to establish the fundamental tradeoff between stored data per node ($\alpha$) and repair bandwidth ($\gamma$) under asymmetric costs.
• Compares the repair cost $C_T$ of the rack model with the static cost model from [4] and the basic model from [3], showing improved cost efficiency.
• Uses numerical evaluation to show that increasing $\tau = C_e/C_c$ reduces repair cost $C_T^1$ and $C_T^2$, with $\eta(\tau)$ decreasing monotonically.

Experimental results

Research questions

• RQ1 How does introducing rack-based communication cost differences affect the fundamental tradeoff between storage and repair bandwidth in distributed systems?
• RQ2 Can a generalized threshold function be derived to model regenerating codes under asymmetric repair costs, such as intra-rack vs. inter-rack bandwidth?
• RQ3 What is the impact of the ratio $\tau = C_e/C_c$ on the repair cost in a two-rack storage model?
• RQ4 How does the proposed rack model compare to the static cost model in [4] and the basic model in [3] in terms of repair cost and bandwidth efficiency?
• RQ5 Under what conditions does the rack model outperform previous models in terms of repair cost, especially when $k > d_1$?

Key findings

• The rack model reduces repair cost compared to the static cost model in [4], particularly when $k > d_1$, due to optimized helper selection based on rack location.
• The repair cost $C_T^1$ and $C_T^2$ decrease monotonically with increasing $\tau = C_e/C_c$, meaning higher inter-rack cost ratios lead to lower overall repair costs.
• The generalized threshold function $f(i)$ successfully extends prior models to handle asymmetric repair costs, enabling accurate tradeoff analysis in realistic settings.
• For $\tau = 1$, the rack model reduces to the basic model in [3], confirming consistency with prior work.
• When $k \leq d_1$, the rack and static cost models behave similarly, but the rack model outperforms the static model when $k > d_1$ in both $\beta_e$ and $\alpha$.
• The model achieves lower repair bandwidth and cost than previous models, with $\eta(\tau) = C_T^{1}(\tau>1)/C_T^{1}(\tau=1)$ decreasing as $\tau$ increases, proving cost efficiency gains.