[Paper Review] Technical Report: Optimistic Execution in Key-Value Store

Duong Nguyen, Aleksey Charapko+2 more|arXiv (Cornell University)|May 25, 2018

Distributed systems and fault tolerance46 references3 citations

TL;DR

This paper proposes a detect-rollback approach to enable applications designed for sequential consistency to safely run on eventually consistent key-value stores like Voldemort. By monitoring correctness predicates (e.g., mutual exclusion in graph processing), the system detects violations and rolls back to a prior consistent state, achieving 50–80% higher throughput with less than 4% monitoring overhead and sub-50ms detection latency in regional networks.

ABSTRACT

Limitations of the CAP theorem imply that if availability is desired in the presence of network partitions, one must sacrifice sequential consistency, a consistency model that is more natural for system design. We focus on the problem of what a designer should do if he/she has an algorithm that works correctly with sequential consistency but is faced with an underlying key-value store that provides a weaker (e.g., eventual or causal) consistency. We propose a detect-rollback based approach: The designer identifies a correctness predicate, say $P$, and continues to run the protocol, as our system monitors $P$. If $P$ is violated (because the underlying key-value store provides a weaker consistency), the system rolls back and resumes the computation at a state where $P$ holds. We evaluate this approach with practical graph applications running on the Voldemort key-value store. Our experiments with deployment on Amazon AWS EC2 instances shows that using eventual consistency with monitoring can provide a $50-80\%$ increase in throughput when compared with sequential consistency. We also show that the overhead of the monitoring itself is low (typically less than 4\%) and the latency of detecting violations is small. In particular, more than $99.9\%$ of violations are detected in less than $50$ milliseconds in regional AWS networks, and in less than $5$ seconds in global AWS networks.

Motivation & Objective

• To address the challenge of running sequential consistency-dependent applications on eventually or causally consistent key-value stores.
• To reduce the need to redesign algorithms for weaker consistency models by enabling safe execution with rollback recovery.
• To evaluate the performance, overhead, and detection latency of predicate monitoring in real-world distributed applications.
• To demonstrate that monitoring can provide significant performance gains while maintaining correctness under network partitions.

Proposed method

• The system monitors a correctness predicate P (e.g., local mutual exclusion in graph processing) using lightweight, non-intrusive monitoring modules.
• Predicate detection is implemented using hybrid vector clocks to reduce false positives and improve efficiency.
• Violations of P trigger a rollback to the most recent consistent snapshot, enabled by existing low-overhead rollback systems like Retroscope.
• The approach supports both linear and semilinear predicates, with algorithms adapted from prior work on distributed predicate detection.
• Monitoring is integrated into the Voldemort key-value store, with experiments conducted on AWS EC2 to evaluate real-world performance.
• The system leverages active replication in Voldemort to allow clients to dynamically switch consistency levels based on monitor feedback.

Experimental results

Research questions

• RQ1 Can applications designed for sequential consistency be safely executed on eventually consistent key-value stores using runtime monitoring and rollback?
• RQ2 What is the performance overhead of monitoring correctness predicates in a distributed key-value store?
• RQ3 How quickly can predicate violations be detected in regional and global network topologies?
• RQ4 To what extent does monitoring improve throughput compared to using sequential consistency?
• RQ5 Can the monitoring framework be used not only for correctness but also for dynamic consistency level adaptation?

Key findings

• The detect-rollback approach achieved a 50–80% increase in throughput when using eventual consistency instead of sequential consistency in graph processing workloads.
• Monitoring overhead was consistently below 4% in normal conditions and under 8% in stressed experiments.
• Over 99.9% of predicate violations were detected within 50 milliseconds in regional AWS networks.
• Global network detection latency remained under 5 seconds, minimizing wasted computation due to rollbacks.
• The number of predicate violations was very low, indicating that the approach is practical even under network partitions.
• Feedback from monitors enabled dynamic adaptation, such as switching to sequential consistency during prolonged network instability.