[Paper Review] Using RDMA for Lock Management

Yeounoh Chung, Erfan Zamanian|arXiv (Cornell University)|Jul 12, 2015

Distributed systems and fault tolerance5 references3 citations

TL;DR

This paper proposes a client-centric lock management design leveraging RDMA to bypass server CPU bottlenecks in distributed lock managers. By enabling clients to perform atomic RDMA operations (e.g., Compare-and-Swap, Fetch-and-Add) directly on lock status, the system achieves up to tenfold higher throughput than traditional centralized designs, especially under high contention.

ABSTRACT

In this work, we aim to evaluate different Distributed Lock Management service designs with Remote Direct Memory Access (RDMA). In specific, we implement and evaluate the centralized and the RDMA-enabled lock manager designs for fast network settings. Experimental results confirms a couple of hypotheses. First, in the traditional centralized lock manager design, CPU is the bottleneck and bypassing CPU on client-to-server communication using RDMA results in better lock service perofrmance. Second, different lock manager designs with RDMA in consideration result in even better performance; we need to re-design lock management system for RDMA and fast networks.

Motivation & Objective

• To address the CPU bottleneck in traditional centralized distributed lock managers (DLMs) under high network load.
• To evaluate whether RDMA-enabled lock management can significantly improve performance in fast networks like InfiniBand.
• To explore re-architecting DLMs with client-side participation in lock management using one-sided RDMA operations.
• To compare traditional server-centric designs with novel client-centric designs that offload lock coordination to clients via RDMA.

Proposed method

• Implement a traditional server-centric DLM using TCP/IP and RDMA send/receive verbs for low-level communication.
• Design a client-centric DLM where clients use RDMA one-sided operations (Compare-and-Swap, Fetch-and-Add, Read, Write) to directly manipulate lock state on the server.
• Use 64-bit lock status words with separate 32-bit fields for exclusive and shared lock ownership, enabling atomic updates via RDMA.
• Implement backoff mechanisms for lock acquisition retries, using RDMA Read to check lock status without modifying it.
• Measure throughput under varying client counts and contention rates using a 7-machine InfiniBand cluster with 40-core processors.
• Compare performance between TCP-based server-centric, RDMA-based server-centric, and client-centric designs.

Experimental results

Research questions

• RQ1 Does bypassing the server CPU via RDMA significantly improve lock service throughput in centralized DLMs?
• RQ2 Can re-architecting the DLM to involve clients in lock coordination using RDMA operations yield better performance than traditional server-centric models?
• RQ3 How do different lock acquisition and release protocols (exclusive vs. shared) perform when implemented with RDMA atomic operations?
• RQ4 What is the impact of client-side backoff strategies on lock acquisition performance when using RDMA?
• RQ5 How does the throughput of client-centric RDMA-based DLM compare to traditional TCP-based and RDMA-based server-centric DLMs under high contention?

Key findings

• In traditional TCP-based centralized DLMs, over 90% of server CPU cycles were consumed by kernel message handling, confirming CPU as a major bottleneck.
• Using RDMA send/receive verbs reduced CPU overhead and improved throughput, but the server remained a performance bottleneck.
• The client-centric design, which offloads lock coordination to clients via RDMA atomic operations, outperformed even the RDMA-optimized server-centric design by an order of magnitude.
• Throughput in the client-centric design scaled significantly better with increasing client count and contention rate, demonstrating superior horizontal scalability.
• Atomic operations like Compare-and-Swap and Fetch-and-Add enabled efficient, low-latency lock acquisition and release without server CPU intervention.
• The use of RDMA Read for status polling during retries avoided unnecessary increments to the shared lock counter, improving correctness and performance.