[Paper Review] Brief Announcement: On Implementing Wear Leveling in Persistent Synchronization Structures

Joseph Izraelevitz, Kansara, Kush+3 more|arXiv (Cornell University)|Jan 1, 2023

Advanced Data Storage Technologies144 citations

TL;DR

This paper presents the first in-depth performance evaluation of Intel's Optane DC Persistent Memory Module, measuring its behavior as a main memory extension and persistent storage layer. It demonstrates that while Optane DC offers significantly lower latency than SSDs and higher capacity than DRAM, its asymmetric bandwidth (6.6 GB/s read vs. 2.3 GB/s write) and higher write latency impact performance—especially in persistent memory workloads. The key finding is that application-level persistence and NVMM-aware file systems can achieve up to 2× performance gains over traditional storage stacks.

ABSTRACT

The last decade has witnessed an explosion of research on persistent memory, which combines the low access latency of dynamic random access memory (DRAM) with the durability of secondary storage. Intel’s implementation of persistent memory, called Optane, comes close to realizing the game-changing potential of persistent memory in terms of performance; however, it also suffers from limited endurance and relies on a proprietary wear leveling mechanism to mitigate memory cell wear-out. The traditional embedded approach to wear leveling, in which the storage device itself maps logical addresses to physical addresses, can be fast and energy-efficient, but it is also relatively inflexible and can lead to missed opportunities for optimization. An alternative school of thought, exemplified by "open channel" solid state drives (SSDs), delegates responsibility for wear leveling to software, where it can be tailored to specific applications. In this research, we consider a hypothetical hardware platform where the same paradigm is applied to the persistent memory device, and ask how the wear leveling mechanism can be co-designed with synchronization structures that generate highly skewed memory access patterns. Building on the recent work of Liu and Golab, we implement an improved wear leveling atomic counter by leveraging hardware transactional memory in a novel way. Our solution is close to optimal with respect to both space complexity and measured performance.

Motivation & Objective

• To evaluate the performance characteristics of Intel's Optane DC Persistent Memory Module (PMM) as a main memory extension and persistent storage device.
• To assess how Optane DC PMM performs across different system modes: Memory (cached) and App Direct (uncached).
• To measure the impact of Optane DC on real-world applications using file systems, persistent memory libraries, and user-space persistence.
• To compare Optane DC performance against DRAM and SSDs in microbenchmarks and production workloads.
• To guide future research by identifying performance bottlenecks and opportunities in persistent memory systems.

Proposed method

• Conducted micro- and macro-benchmarks on a 3 TB Optane DC PMM system with dual Xeon Scalable processors.
• Evaluated performance in two modes: Memory (cached) mode with DRAM cache and App Direct (uncached) mode with direct user-space access.
• Measured latency, bandwidth, and throughput across a range of workloads including PMemKV, WHISPER, and production applications (e.g., RocksDB, Redis, MongoDB).
• Used multiple file systems: Ext4, XFS, NOVA, and DAX-optimized variants, comparing performance with and without NVMM awareness.
• Measured performance using both kernel-level file system I/O and user-space persistent memory programming with clwb and fence instructions.
• Normalized results against DRAM-based PMem (PM-LDRAM) and RDRAM to isolate the impact of Optane DC’s performance characteristics.

Experimental results

Research questions

• RQ1 How does Optane DC PMM performance compare to DRAM and SSDs in terms of latency and bandwidth across different access patterns?
• RQ2 What is the performance impact of using Optane DC in cached (Memory) vs. uncached (App Direct) modes on real-world applications?
• RQ3 To what extent can NVMM-aware file systems and user-space persistent memory programming improve application performance on Optane DC?
• RQ4 How do persistent data structures and workloads with high write frequency perform on Optane DC compared to DRAM?
• RQ5 How do software stack choices (file system, DAX, user-space persistence) affect end-to-end application throughput on Optane DC?

Key findings

• Optane DC PMM has a read bandwidth of 6.6 GB/s and write bandwidth of 2.3 GB/s, showing significant asymmetry, with a 346 ns access latency higher than DRAM but lower than SSDs.
• In cached (Memory) mode, applications with small memory footprints see minimal performance impact, while larger-footprint applications benefit from increased memory capacity.
• In App Direct (uncached) mode, using the NOVA-relaxed NVMM file system improved Kyoto Cabinet performance by nearly 2× compared to traditional file systems.
• User-space persistent versions of applications like RocksDB achieved up to 2× higher performance than their file system counterparts, demonstrating the benefit of bypassing kernel and file system overhead.
• WHISPER benchmark results show that Optane DC increases application execution time by an average of 24% compared to DRAM, with the largest performance gap in write-intensive persistent data structures.
• The performance gap between Optane DC and DRAM is most pronounced in write-heavy workloads, with PMemKV showing up to 2.05× higher write latency on Optane DC than on PM-LDRAM.