[Paper Review] A Robust Fault-Tolerant and Scalable Cluster-wide Deduplication for Shared-Nothing Storage Systems

Awais Khan, Chang-Gyu Lee+3 more|arXiv (Cornell University)|Mar 21, 2018

Advanced Data Storage Technologies13 references3 citations

TL;DR

This paper proposes a scalable, fault-tolerant, and cluster-wide deduplication framework for shared-nothing storage systems like Ceph, using content-based fingerprinting and a tagged consistency mechanism to eliminate metadata bottlenecks and ensure transactional integrity. The approach achieves high storage efficiency with minimal performance overhead and robust failure recovery.

ABSTRACT

Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared- nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.

Motivation & Objective

• Address the lack of scalable and fault-tolerant deduplication in shared-nothing storage systems (SN-SS), which traditionally suffer from metadata bottlenecks and consistency issues under failures.
• Overcome the limitations of centralized deduplication metadata servers and inefficient DB-sharding approaches that cause broadcasted lookups and high I/O overhead during storage rebalancing.
• Ensure transactional consistency and garbage chunk detection without relying on expensive journaling or checkpointing mechanisms.
• Design a system that preserves the shared-nothing architecture while enabling cluster-wide duplicate detection and efficient metadata management.

Proposed method

• Use content-based fingerprints of data chunks to determine their global location in the cluster, eliminating the need for broadcasted metadata lookups across all shards.
• Implement a distributed deduplication metadata shard (DM-Shard) that is co-located with object storage servers, preserving the shared-nothing property and enabling horizontal scalability.
• Introduce an asynchronous tagged consistency mechanism using commit flags per chunk or object to track transaction states without blocking I/O or requiring transaction locks.
• Leverage CRUSH-based data placement in Ceph to ensure uniform load distribution and efficient chunk location resolution via fingerprints.
• Design the system to automatically detect and clean up garbage chunks from failed transactions using the tagged consistency model.
• Integrate the solution into Ceph and evaluate it using real testbeds with varying workloads and failure scenarios.

Experimental results

Research questions

• RQ1 How can deduplication metadata be scaled and decentralized in shared-nothing storage systems without introducing central bottlenecks or excessive I/O overhead?
• RQ2 What mechanisms can ensure transactional consistency and fault tolerance in a distributed deduplication system under sudden storage server failures?
• RQ3 How does content-based chunk fingerprinting improve metadata lookup efficiency and reduce broadcasted queries in a cluster-wide deduplication setting?
• RQ4 To what extent does the proposed tagged consistency mechanism reduce performance overhead compared to traditional lock-based or logging-based consistency models?
• RQ5 How does the proposed cluster-wide deduplication approach compare to local disk-based deduplication in terms of storage efficiency as the number of disks increases?

Key findings

• The cluster-wide deduplication approach achieves 85% storage space savings across 1 to 8 disks, while disk-based deduplication degrades to 61% savings with 8 disks due to lack of cross-disk duplicate detection.
• With 32 client threads, the central deduplication approach degrades to 200 MB/s bandwidth due to metadata server contention, whereas the proposed cluster-wide approach maintains scalability and improves bandwidth with increasing concurrency.
• The asynchronous tagged consistency mechanism incurs negligible performance overhead compared to chunk- and object-level consistency models, which suffer from 15%+ performance degradation due to transaction locks.
• The system demonstrates high robustness under sudden server failures, correctly identifying and handling garbage chunks without additional monitoring or journaling.
• Scalability analysis shows that the proposed system outperforms central deduplication under high I/O contention, with bandwidth increasing as client threads scale, thanks to distributed metadata and load-aware CRUSH placement.
• The solution eliminates the need for centralized metadata servers and avoids broadcasted metadata lookups, enabling efficient and scalable cluster-wide deduplication in shared-nothing architectures.