[Paper Review] Write-Optimized and Consistent RDMA-based NVM Systems

Xinxin Liu, Yu Hua+2 more|arXiv (Cornell University)|Jun 18, 2019

Advanced Memory and Neural Computing26 references4 citations

TL;DR

Erda is a zero-copy, log-structured memory design that ensures Remote Data Atomicity (RDA) for one-sided RDMA writes to NVM without extra network round-trips, remote CPU involvement, or double NVM writes. By using CRC checksums for incomplete write detection and 8-byte atomic metadata updates, Erda reduces NVM writes by ~50% while improving throughput and reducing latency compared to redo logging and read-after-write schemes.

ABSTRACT

In order to deliver high performance in cloud computing, we generally exploit and leverage RDMA (Remote Direct Memory Access) in networking and NVM (Non-Volatile Memory) in end systems. Due to no involvement of CPU, one-sided RDMA becomes efficient to access the remote memory, and NVM technologies have the strengths of non-volatility, byte-addressability and DRAM-like latency. In order to achieve end-to-end high performance, many efforts aim to synergize one-sided RDMA and NVM. Due to the need to guarantee Remote Data Atomicity (RDA), we have to consume extra network round-trips, remote CPU participation and double NVM writes. In order to address these problems, we propose a zero-copy log-structured memory design for Efficient Remote Data Atomicity, called Erda. In Erda, clients directly transfer data to the destination address at servers via one-sided RDMA writes without redundant copy and remote CPU consumption. To detect the incompleteness of fetched data, we verify a checksum without client-server coordination. We further ensure metadata consistency by leveraging an 8-byte atomic update in the hash table, which also contains the address information for the stale data. When a failure occurs, the server properly restores to a consistent version. Experimental results show that compared with Redo Logging (a CPU involvement scheme) and Read After Write (a network dominant scheme), Erda reduces NVM writes approximately by 50%, as well as significantly improves throughput and decreases latency.

Motivation & Objective

• To address high network overheads, CPU consumption, and double NVM writes in RDMA-based NVM systems that arise when guaranteeing Remote Data Atomicity (RDA).
• To enable one-sided RDMA writes to NVM with end-to-end consistency and low latency, avoiding remote CPU coordination.
• To reduce NVM write amplification by eliminating redundant copies and persistent logging overheads.
• To ensure data consistency after failures through checksum-based detection and atomic metadata updates.
• To achieve high performance in cloud workloads by synergizing one-sided RDMA and NVM without sacrificing durability or atomicity.

Proposed method

• Erda uses a log-structured design where clients directly write data to server NVM via one-sided RDMA, bypassing intermediate buffers and server CPU.
• Each data object includes a CRC checksum to detect incomplete or corrupted writes during subsequent read operations.
• Metadata updates are performed using 8-byte atomic writes to a hash table, which stores both the new data address and the previous version’s address for recovery.
• Upon detecting an incomplete write via checksum verification, clients re-read the previous version using the address stored in the hash table.
• Servers detect inconsistency and restore to a consistent state using the stored previous version and atomic metadata updates.
• The design avoids extra network round-trips and eliminates double NVM writes by not requiring logging or coordination for atomicity.

Experimental results

Research questions

• RQ1 How can Remote Data Atomicity (RDA) be guaranteed in one-sided RDMA-based NVM systems without incurring extra network round-trips?
• RQ2 Can remote data atomicity be achieved without remote CPU involvement or redundant data copies?
• RQ3 What mechanisms can reduce NVM write amplification in RDMA-NVM systems while maintaining consistency?
• RQ4 How can incomplete or inconsistent writes be detected and recovered from efficiently in a zero-copy, remote-direct environment?
• RQ5 Can a log-structured design with checksums and atomic metadata updates provide both performance and consistency in RDMA-NVM systems?

Key findings

• Erda reduces NVM writes by approximately 50% compared to redo logging and read-after-write schemes by eliminating redundant writes and logging.
• Throughput is significantly improved due to reduced network round-trips and CPU overhead from remote operations.
• Latency is decreased because data are written directly to NVM without intermediate buffering or CPU processing.
• The system achieves remote data atomicity through checksum verification and 8-byte atomic metadata updates, ensuring consistency after failures.
• Erda avoids double NVM writes by not requiring persistent logging or buffer copies, improving NVM endurance and performance.
• The source code is publicly released, enabling reproducibility and further system integration.