[Paper Review] On Atomic Registers and Randomized Consensus in M&M Systems

Vassos Hadzilacos, Xing Hu+1 more|arXiv (Cornell University)|Jun 1, 2019

Distributed systems and fault tolerance26 references4 citations

TL;DR

This paper presents optimal algorithms for implementing atomic single-writer multi-reader (SWMR) registers and randomized consensus in message-and-memory (m&m) systems, which combine message passing and shared memory (e.g., RDMA). It establishes that the fault tolerance of these algorithms depends on the topology of the shared-memory graph, and proves that m RDMA connections are both necessary and sufficient to tolerate m process crashes, generalizing classic results from pure message-passing systems.

ABSTRACT

Motivated by recent distributed systems technology, Aguilera et al. introduced a hybrid model of distributed computing, called message-and-memory model or m&m model for short [1]. In this model, processes can communicate by message passing and also by accessing some shared memory (e.g., through some RDMA connections). We first consider the basic problem of implementing an atomic single-writer multi-reader (SWMR) register shared by all the processes in m&m systems. Specifically, we give an algorithm that implements such a register in m&m systems and show that it is optimal in the number of process crashes that it can tolerate. This generalizes the well-known implementation of an atomic SWMR register in a pure message-passing system [5]. We then combine our register implementation for m&m systems with the well-known randomized consensus algorithm of Aspnes and Herlihy [4], and obtain a randomized consensus algorithm for m&m systems that is also optimal in the number of process crashes that it can tolerate. Finally, we determine the minimum number of RDMA connections that is sufficient to implement a SWMR register, or solve randomized consensus, in an m&m system with t process crashes, for any given t.

Motivation & Objective

• To design an optimal atomic SWMR register implementation in m&m systems that tolerates the maximum number of process crashes.
• To extend randomized consensus algorithms to m&m systems with optimal fault tolerance by leveraging shared memory via RDMA.
• To determine the minimum number of RDMA connections required to achieve any desired fault tolerance in uniform m&m systems.
• To characterize fault tolerance in terms of shared-memory graph topology, enabling efficient system design with limited connections per process.

Proposed method

• Proposes a new algorithm for implementing atomic SWMR registers in m&m systems, generalizing the ABD algorithm for pure message-passing systems.
• Uses the shared-memory graph G to model RDMA connections, where nodes are processes and edges represent shared-memory access.
• Applies the randomized consensus algorithm of Aspnes and Herlihy, substituting atomic registers with the proposed m&m register implementation.
• Proves fault tolerance bounds using graph-theoretic analysis, particularly focusing on the square of the graph G² and connectivity properties.
• Derives necessary and sufficient conditions for fault tolerance by analyzing disjoint node sets and their neighborhood intersections in G.
• Establishes optimality through contradiction: shows that exceeding the derived fault tolerance bound is impossible under the m&m model.

Experimental results

Research questions

• RQ1 What is the maximum number of process crashes that can be tolerated when implementing an atomic SWMR register in an m&m system?
• RQ2 Can randomized consensus be solved with optimal fault tolerance in m&m systems by combining shared memory and message passing?
• RQ3 What is the minimum number of RDMA connections required to achieve m-fault tolerance in a uniform m&m system?
• RQ4 How does the topology of the shared-memory graph G affect the fault tolerance of register and consensus implementations?
• RQ5 Can m&m systems with bounded-degree graphs (e.g., limited RDMA connections per process) achieve higher fault tolerance than pure message-passing systems?

Key findings

• For any m&m system with shared-memory graph G, the maximum number of tolerable crashes t_G is determined by the graph’s topology, specifically the connectivity of G².
• An m&m system with m RDMA connections can tolerate up to m process crashes, and this is both necessary and sufficient for optimal fault tolerance.
• The star graph topology achieves optimal fault tolerance with minimal connections, where one central node connects to all others, enabling t_G = n-1 tolerance.
• For k=2, a cycle-based graph achieves t_G = ⌈n/2⌉ + 1 fault tolerance, outperforming pure message-passing systems.
• When no RDMA connections are used, fault tolerance is limited to ⌈n/2⌉ - 1 crashes, but this bound is surpassed with as few as m > ⌈n/2⌉ - 1 connections.
• The proposed consensus algorithm is correct under a strong adversary even when using regular registers, due to the specific properties of the Aspnes-Herlihy algorithm.