[Paper Review] Bipartisan Paxos: A Modular State Machine Replication Protocol

Michael Whittaker, Neil Giridharan+3 more|arXiv (Cornell University)|Feb 29, 2020

Distributed systems and fault tolerance26 references4 citations

TL;DR

Bipartisan Paxos (BPaxos) is a modular, multi-leader state machine replication protocol that decouples protocol components to enable independent scaling of high-throughput elements like leaders and proposers. By isolating scalable and non-scalable components, BPaxos achieves roughly double the peak throughput of EPaxos while significantly improving protocol simplicity through modular correctness proofs.

ABSTRACT

There is no shortage of state machine replication protocols. From Generalized Paxos to EPaxos, a huge number of replication protocols have been proposed that achieve high throughput and low latency. However, these protocols all have two problems. First, they do not scale. Many protocols actually slow down when you scale them, instead of speeding up. For example, increasing the number of MultiPaxos acceptors increases quorum sizes and slows down the protocol. Second, they are too complicated. This is not a secret; state machine replication is notoriously difficult to understand. In this paper, we tackle both problems with a single solution: modularity. We present Bipartisan Paxos (BPaxos), a modular state machine replication protocol. Modularity yields high throughput via scaling. We note that while many replication protocol components do not scale, some do. By modularizing BPaxos, we are able to disentangle the two and scale the bottleneck components to increase the protocol's throughput. Modularity also yields simplicity. BPaxos is divided into a number of independent modules that can be understood and proven correct in isolation.

Motivation & Objective

• To address the scalability and complexity problems inherent in existing state machine replication protocols like MultiPaxos and EPaxos.
• To enable high throughput through independent scaling of protocol components, particularly leaders and proposers, which are embarrassingly parallel.
• To improve protocol understandability by decomposing the system into independent, provably correct modules.
• To demonstrate that modularity enables both performance scaling and reduced implementation complexity compared to tightly coupled protocols.

Proposed method

• Decomposing the state machine replication protocol into modular components: leaders, proposers, acceptors, replicas, and a dependency service.
• Using Paxos as a black-box consensus module to inherit safety properties without reimplementing consensus logic.
• Enabling independent horizontal scaling of leaders and proposers, which are the primary throughput bottlenecks.
• Leveraging batching by aggregating client commands into single vertices to amortize overheads across leaders and proposers.
• Designing the protocol so that replicas process smaller, batched graphs, reducing topological sort time and improving execution efficiency.
• Disaggregating components to allow independent deployment and scaling, avoiding the performance bottlenecks of monolithic designs like EPaxos.

Experimental results

Research questions

• RQ1 Can modularity in state machine replication protocols enable independent scaling of high-throughput components without compromising correctness?
• RQ2 Does modular decomposition significantly reduce protocol complexity compared to tightly coupled protocols like EPaxos or GPaxos?
• RQ3 Can a modular protocol achieve higher throughput than state-of-the-art protocols like EPaxos through scalable component deployment?
• RQ4 How does batching interact with modular design to further improve throughput and reduce latency?
• RQ5 Can well-established protocols like Paxos be reused as modular components to ensure safety and simplify verification?

Key findings

• BPaxos achieves roughly double the peak throughput of EPaxos by independently scaling leaders and proposers, which are the primary throughput bottlenecks.
• With a batch size of 1000, BPaxos achieves a peak throughput of approximately 500,000 commands per second and a median latency of around 200 ms under optimal conditions.
• The modular design allows each component—such as the consensus module, dependency service, and replicas—to be understood and verified in isolation, improving protocol maintainability and correctness.
• BPaxos outperforms single-leader protocols like MultiPaxos and Raft by eliminating the single-leader bottleneck through multi-leader architecture.
• The protocol’s decoupled design enables efficient batching, where overheads are amortized across leaders and proposers, and replica graph processing is accelerated due to fewer, larger vertices.
• BPaxos avoids the complexity of fast-path optimizations found in EPaxos and Caesar, reducing the risk of subtle bugs such as those discovered in EPaxos’ consensus implementation after six years.