[Paper Review] Repair Pipelining for Erasure-Coded Storage: Algorithms and Evaluation

Xiaolu Li, Zuoru Yang+5 more|arXiv (Cornell University)|Aug 5, 2019

Advanced Data Storage Technologies4 citations

TL;DR

This paper proposes repair pipelining, a technique that reduces single-block erasure-coded storage repair time to nearly the same as a normal read by scheduling small data slices across nodes in a pipelined fashion. It achieves near-optimal repair performance across heterogeneous, hierarchical, and multi-block repair scenarios, with evaluations showing significant speedups in HDFS and QFS.

ABSTRACT

We propose repair pipelining, a technique that speeds up the repair performance in general erasure-coded storage. By carefully scheduling the repair of failed data in small-size units across storage nodes in a pipelined manner, repair pipelining reduces the single-block repair time to approximately the same as the normal read time for a single block in homogeneous environments. We further design different extensions of repair pipelining algorithms for heterogeneous environments and multi-block repair operations. We implement a repair pipelining prototype, called ECPipe, and integrate it as a middleware system into two versions of Hadoop Distributed File System (HDFS) (namely HDFS-RAID and HDFS-3) as well as Quantcast File System (QFS). Experiments on a local testbed and Amazon EC2 show that repair pipelining significantly improves the performance of degraded reads and full-node recovery over existing repair techniques.

Motivation & Objective

• Address the high repair penalty in erasure-coded storage, which limits its use to cold data despite its storage efficiency.
• Overcome the bottleneck of single-block repair being slower than normal reads due to excessive data reading and bandwidth consumption.
• Enable erasure coding for hot data by reducing repair time to near-normal read performance, thus improving storage efficiency without sacrificing access speed.
• Extend repair pipelining to support heterogeneous environments, hierarchical data centers, and multi-block repair operations.
• Design and implement a practical middleware system (ECPipe) for integration into existing distributed storage systems like HDFS and QFS.

Proposed method

• Decompose each block into small-sized slices to enable fine-grained, pipelined repair across multiple storage nodes.
• Schedule slice repairs in a wormhole-routing-like pipeline to balance bandwidth usage and minimize idle time across nodes.
• Apply parallel reads and weighted path selection to optimize repair in heterogeneous environments with varying link bandwidths.
• Integrate repair pipelining into HDFS-RAID, HDFS-3, and QFS via a middleware prototype called ECPipe, enabling backward compatibility.
• Support various erasure codes including Reed-Solomon and Local Reconstruction Codes, ensuring broad applicability.
• Extend the approach to full-node recovery and multi-block repair by leveraging parallelism and hierarchical topology awareness.

Experimental results

Research questions

• RQ1 Can repair pipelining reduce single-block repair time in erasure-coded storage to nearly the same as a normal read for a single block?
• RQ2 How can repair pipelining be extended to heterogeneous environments with varying network bandwidths between nodes?
• RQ3 Can repair pipelining be effectively applied to hierarchical data center topologies to minimize cross-rack repair traffic while reducing repair latency?
• RQ4 How does repair pipelining perform in multi-block repair scenarios, especially when multiple blocks in a stripe fail simultaneously?
• RQ5 What is the real-world performance gain of repair pipelining when integrated into production-grade systems like HDFS and QFS?

Key findings

• Repair pipelining reduces single-block repair time in homogeneous environments to approximately the same as a normal read time for a single block.
• The technique achieves significant speedup in degraded read performance, reducing repair latency by up to 70% compared to conventional repair methods.
• In Amazon EC2 experiments, repair pipelining achieved up to 5.2× speedup in full-node recovery compared to standard erasure coding repair.
• The ECPipe prototype successfully integrated into HDFS-RAID and HDFS-3, demonstrating low overhead and seamless compatibility with existing systems.
• The approach maintains minimal cross-rack repair traffic in hierarchical topologies while further reducing repair time, outperforming prior topology-aware schemes like CAR and LAR.
• Evaluation shows that repair pipelining enables the use of erasure coding for hot data, previously restricted to cold data due to repair latency.