[Paper Review] Performance and Fault Tolerance in the StoreTorrent Parallel Filesystem

Federico D. Sacerdoti|arXiv (Cornell University)|Jan 21, 2010

Advanced Data Storage Technologies21 references4 citations

TL;DR

StoreTorrent is a parallel filesystem designed for efficient handling of small data records on commodity hardware, using application-OS pipelining and peer-to-peer replica index sharing to boost performance and fault tolerance. It achieved 1.69 GB/s for parallel small writes and 8.47 GB/s for network reads across 70 nodes, outperforming PVFS and Gluster in both benchmarks.

ABSTRACT

With a goal of supporting the timely and cost-effective analysis of Terabyte datasets on commodity components, we present and evaluate StoreTorrent, a simple distributed filesystem with integrated fault tolerance for efficient handling of small data records. Our contributions include an application-OS pipelining technique and metadata structure to increase small write and read performance by a factor of 1-10, and the use of peer-to-peer communication of replica-location indexes to avoid transferring data during parallel analysis even in a degraded state. We evaluated StoreTorrent, PVFS, and Gluster filesystems using 70 storage nodes and 560 parallel clients on an 8-core/node Ethernet cluster with directly attached SATA disks. StoreTorrent performed parallel small writes at an aggregate rate of 1.69 GB/s, and supported reads over the network at 8.47 GB/s. We ported a parallel analysis task and demonstrate it achieved parallel reads at the full aggregate speed of the storage node local filesystems.

Motivation & Objective

• To enable cost-effective, timely analysis of terabyte-scale datasets using commodity hardware.
• To address performance bottlenecks in small record I/O for distributed filesystems.
• To enhance fault tolerance without sacrificing I/O performance during degraded states.
• To eliminate data transfer during parallel analysis by sharing replica location indexes via peer-to-peer communication.
• To demonstrate end-to-end performance gains in real-world parallel workloads.

Proposed method

• Application-OS pipelining to overlap application-level I/O with OS-level system calls, reducing latency for small writes.
• A specialized metadata structure optimized for fast lookup and update of small record metadata.
• Peer-to-peer exchange of replica location indexes among storage nodes to enable data access without data transfer during parallel reads.
• Replication of metadata and data across nodes to ensure availability and fault tolerance.
• Integration of the filesystem with a 70-node, 8-core Ethernet cluster using directly attached SATA disks.
• Benchmarking against PVFS and Gluster using 560 parallel clients to evaluate write and read performance.

Experimental results

Research questions

• RQ1 Can application-OS pipelining significantly improve small record write performance in a distributed filesystem?
• RQ2 Can peer-to-peer sharing of replica location indexes eliminate data transfer during parallel analysis, even in degraded states?
• RQ3 What is the achievable aggregate I/O performance of a commodity-based parallel filesystem for small files?
• RQ4 How does StoreTorrent compare to existing systems like PVFS and Gluster in terms of write and read throughput?
• RQ5 Can a fault-tolerant filesystem maintain high performance during node failures or network degradation?

Key findings

• StoreTorrent achieved an aggregate parallel small write performance of 1.69 GB/s across 70 nodes, demonstrating significant performance gains over baseline systems.
• The system sustained network-based parallel reads at 8.47 GB/s, approaching the aggregate local filesystem bandwidth of the storage nodes.
• The use of peer-to-peer replica index sharing allowed parallel analysis to proceed without data transfer, even when some nodes were degraded or failed.
• The application-OS pipelining and optimized metadata structure contributed to 1-10x performance improvements in small I/O operations.
• A ported parallel analysis task achieved data read rates at the full aggregate speed of the local storage subsystems, confirming end-to-end performance efficiency.
• StoreTorrent outperformed both PVFS and Gluster in both write and read workloads under the same test conditions.