[Paper Review] Scaling Replicated State Machines with Compartmentalization [Technical Report]

Michael Whittaker, Ailidani Ailijiang+7 more|arXiv (Cornell University)|Dec 31, 2020

Distributed systems and fault tolerance44 references4 citations

TL;DR

This paper introduces compartmentalization, a technique to eliminate throughput bottlenecks in state machine replication protocols like MultiPaxos by decoupling and independently scaling distinct components—such as command sequencing and broadcasting—resulting in a 6× throughput improvement on write-only workloads and 16× on mixed read-write workloads without specialized hardware or strong assumptions.

ABSTRACT

State machine replication protocols, like MultiPaxos and Raft, are a critical component of many distributed systems and databases. However, these protocols offer relatively low throughput due to several bottlenecked components. Numerous existing protocols fix different bottlenecks in isolation but fall short of a complete solution. When you fix one bottleneck, another arises. In this paper, we introduce compartmentalization, the first comprehensive technique to eliminate state machine replication bottlenecks. Compartmentalization involves decoupling individual bottlenecks into distinct components and scaling these components independently. Compartmentalization has two key strengths. First, compartmentalization leads to strong performance. In this paper, we demonstrate how to compartmentalize MultiPaxos to increase its throughput by 6x on a write-only workload and 16x on a mixed read-write workload. Unlike other approaches, we achieve this performance without the need for specialized hardware. Second, compartmentalization is a technique, not a protocol. Industry practitioners can apply compartmentalization to their protocols incrementally without having to adopt a completely new protocol.

Motivation & Objective

• To address the persistent performance bottleneck in state machine replication protocols like MultiPaxos and Raft, which suffer from overloaded leaders due to monolithic responsibilities.
• To overcome the limitation of existing protocols that fix only one bottleneck at a time, leading to new bottlenecks emerging.
• To enable high-throughput state machine replication without relying on specialized hardware, perfect failure detectors, clock synchrony, or key-partitioned state machines.
• To provide a general, incremental technique that allows practitioners to enhance existing protocols without replacing them with entirely new ones.

Proposed method

• Decompose the monolithic leader role in MultiPaxos into distinct components: command sequencing and message broadcasting.
• Introduce proxy leaders to handle message broadcasting independently from the sequencer, enabling horizontal scaling of the broadcast phase.
• Scale the number of proxy leaders to eliminate bottlenecks in the broadcasting phase, which is embarrassingly parallel and thus highly parallelizable.
• Maintain protocol correctness and consistency by ensuring all components adhere to the original MultiPaxos safety guarantees through proper coordination and quorum mechanisms.
• Apply the compartmentalization technique incrementally to existing protocols, allowing deployment without full protocol migration.
• Use a modular, component-based architecture where each function (sequencing, broadcasting, etc.) runs in isolation, enabling independent optimization and scaling.

Experimental results

Research questions

• RQ1 Can state machine replication protocols be scaled effectively without relying on specialized hardware or strong system assumptions?
• RQ2 Does decoupling monolithic components of a protocol like MultiPaxos lead to measurable throughput improvements?
• RQ3 Can the performance gains from decoupling be achieved incrementally, without replacing the entire protocol stack?
• RQ4 How does compartmentalization compare to existing high-throughput protocols like Harmonia or FLAIR in terms of assumptions and performance?
• RQ5 What is the trade-off between throughput and latency when applying compartmentalization to state machine replication?

Key findings

• Compartmentalized MultiPaxos achieves a 6× throughput improvement on a write-only workload using 6.66× the number of machines.
• On a 90% read, 10% write workload, the protocol achieves a 16× throughput improvement using 4.33× the number of machines.
• The performance gains are achieved without requiring specialized hardware, perfect failure detectors, clock synchrony, or uniform data access patterns.
• The technique is general and can be incrementally applied to existing protocols, enabling practitioners to upgrade their systems without full protocol replacement.
• The approach eliminates both write and read bottlenecks by decoupling sequencing and broadcasting, with no single component becoming a performance limiter.
• Despite adding two extra network delays (from 4 to 6), the protocol remains viable for wide-area deployments due to the substantial throughput gains.