[Paper Review] Harmonia: Near-Linear Scalability for Replicated Storage with In-Network Conflict Detection

Hang Zhu, Zhihao Bai+5 more|arXiv (Cornell University)|Apr 18, 2019

Distributed systems and fault tolerance46 references4 citations

TL;DR

Harmonia is a replicated storage system that achieves near-linear scalability with strong consistency by offloading read-write conflict detection to programmable network switches. By tracking dirty objects in the data plane at line rate, it enables any replica to serve reads for non-conflicting objects, boosting throughput up to 10× for replication factors of 10 without performance overhead.

ABSTRACT

Distributed storage employs replication to mask failures and improve availability. However, these systems typically exhibit a hard tradeoff between consistency and performance. Ensuring consistency introduces coordination overhead, and as a result the system throughput does not scale with the number of replicas. We present Harmonia, a replicated storage architecture that exploits the capability of new-generation programmable switches to obviate this tradeoff by providing near-linear scalability without sacrificing consistency. To achieve this goal, Harmonia detects read-write conflicts in the network, which enables any replica to serve reads for objects with no pending writes. Harmonia implements this functionality at line rate, thus imposing no performance overhead. We have implemented a prototype of Harmonia on a cluster of commodity servers connected by a Barefoot Tofino switch, and have integrated it with Redis. We demonstrate the generality of our approach by supporting a variety of replication protocols, including primary-backup, chain replication, Viewstamped Replication, and NOPaxos. Experimental results show that Harmonia improves the throughput of these protocols by up to 10X for a replication factor of 10, providing near-linear scalability up to the limit of our testbed.

Motivation & Objective

• To address the performance bottleneck in replicated storage systems where increasing replicas degrades throughput due to coordination overhead.
• To enable near-linear scalability for read-heavy workloads without sacrificing strong consistency (linearizability).
• To eliminate the performance penalty of traditional conflict detection by moving it into the network data plane using programmable switches.
• To support a wide range of replication protocols—including primary-backup, chain replication, Viewstamped Replication, and NOPaxos—without modifying their core logic.
• To demonstrate that in-network processing can enable scalable, consistent, and low-overhead read distribution across replicas.

Proposed method

• Deploy a programmable switch (Barefoot Tofino) in the data path to monitor all read and write requests to the storage cluster.
• Implement a conflict detection module in the switch’s data plane that maintains a real-time dirty set of objects with pending writes.
• Use the switch to inspect each read request and determine if it conflicts with any in-progress write; if not, redirect it to any replica.
• Ensure the switch maintains a consistent view of the last-committed point and the dirty set using switch state synchronization across failovers.
• Integrate Harmonia with existing storage systems (e.g., Redis) by extending the client and server logic to support switch-directed routing.
• Leverage the switch’s line-rate processing capability to perform conflict checks with near-zero latency and overhead.

Experimental results

Research questions

• RQ1 Can in-network conflict detection enable near-linear scalability in replicated storage without sacrificing strong consistency?
• RQ2 Can programmable switches efficiently track and enforce consistency for a large number of objects with minimal memory and latency?
• RQ3 Can the conflict detection mechanism be generalized across diverse replication protocols such as primary-backup and quorum-based systems?
• RQ4 Does offloading conflict detection to the network eliminate the performance bottleneck seen in traditional replicated systems as replica count increases?
• RQ5 Can the system maintain high throughput and low latency during switch failover or recovery?

Key findings

• Harmonia achieves up to 10× higher throughput compared to traditional replication protocols at a replication factor of 10, demonstrating near-linear scalability.
• The system maintains linearizability by ensuring that only reads to non-dirty objects are redirected to replicas, preventing stale or uncommitted data from being served.
• Conflict detection is performed at line rate in the switch data plane with negligible performance overhead, enabling real-time, scalable routing decisions.
• The switch-based dirty set tracking remains consistent across failures, allowing the system to restore full throughput within seconds after a switch restart.
• Harmonia is compatible with multiple replication protocols, including NOPaxos, Viewstamped Replication, chain replication, and primary-backup, without modifying their core logic.
• The prototype implementation on a commodity cluster with a Barefoot Tofino switch confirms that the approach is practical and efficient in real-world deployments.