[Paper Review] Assise: Performance and Availability via NVM Colocation in a Distributed File System

Thomas E. Anderson, Marco Canini+7 more|arXiv (Cornell University)|Oct 7, 2019

Advanced Data Storage Technologies50 references8 citations

TL;DR

Assise is a distributed file system that leverages node-local non-volatile memory (NVM) to achieve sub-second failover, low tail latency, and strong consistency by colocating computation and storage. It uses a crash-consistent, replicated coherence protocol (CC-NVM) to manage NVM caches with per-I/O operation granularity, outperforming Ceph, NFS, and Octopus by up to 56× in throughput and 103× in failover time while ensuring linearizability and prefix semantics.

ABSTRACT

The adoption of very low latency persistent memory modules (PMMs) upends the long-established model of disaggregated file system access. Instead, by colocating computation and PMM storage, we can provide applications much higher I/O performance, sub-second application failover, and strong consistency. To demonstrate this, we built the Assise distributed file system, based on a persistent, replicated coherence protocol for managing a set of server-colocated PMMs as a fast, crash-recoverable cache between applications and slower disaggregated storage, such as SSDs. Unlike disaggregated file systems, Assise maximizes locality for all file IO by carrying out IO on colocated PMM whenever possible and minimizes coherence overhead by maintaining consistency at IO operation granularity, rather than at fixed block sizes. We compare Assise to Ceph/Bluestore, NFS, and Octopus on a cluster with Intel Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as LevelDB, Postfix, and FileBench. We find that Assise improves write latency up to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x better than its counterparts, while providing stronger consistency semantics. Assise promises to beat the MinuteSort world record by 1.5x.

Motivation & Objective

• Address the performance and availability limitations of disaggregated file systems in the era of low-latency persistent memory (NVM).
• Overcome high system call overhead, long recovery times, and block-granularity coherence in traditional file systems when using NVM.
• Enable high-performance, crash-consistent, and highly available file I/O without requiring application-level API changes.
• Design a scalable, low-latency distributed file system that leverages local, socket-local, and network-local NVM for caching and replication.
• Achieve strong consistency with minimal coherence overhead by managing consistency at the I/O operation level rather than fixed block sizes.

Proposed method

• Colocate application processes with persistent memory modules (PMMs) to eliminate network latency for I/O operations.
• Implement CC-NVM, a crash-consistent, replicated cache coherence layer that ensures prefix semantics and linearizability via lease delegation and logging.
• Use RDMA and DMA to enforce write ordering across remote and cross-socket NVM, enabling strong consistency without kernel intervention.
• Support kernel-bypass I/O to authorized local and remote NVM regions to minimize tail latency.
• Enable optimistic mode with asynchronous chain replication for low-write-latency persistence with prefix crash consistency.
• Deploy reserve replicas as network-local NVM caches that become cache replicas in cascaded failures, ensuring near-instant failover.

Experimental results

Research questions

• RQ1 Can colocated NVM significantly reduce I/O latency and failover time in distributed file systems compared to disaggregated models?
• RQ2 How can strong consistency be maintained with minimal coherence overhead when using NVM as a high-speed, persistent cache?
• RQ3 To what extent can local lease management and process-local caching improve scalability and reduce tail latency in file system workloads?
• RQ4 Can a crash-consistent, replicated NVM cache layer (CC-NVM) provide both high performance and high availability without sacrificing consistency?
• RQ5 How does Assise’s design scale under real-world workloads such as LevelDB, Postfix, and FileBench compared to Ceph, NFS, and Octopus?

Key findings

• Assise reduces write latency by up to 22× and increases throughput by up to 56× compared to Ceph/Bluestore on workloads including LevelDB and FileBench.
• Application failover time is improved by up to 103× due to local recovery of NVM caches using crash-consistent replicas.
• Assise scales linearly with the number of processes until NVM write bandwidth becomes the bottleneck, achieving 6× better scalability than Ceph.
• The sharded configuration (Assise-sharded) improves performance by up to 20% over round-robin due to localized consistency management, even with shared directories.
• Private directory sharding (Assise-private) achieves performance comparable to sharding-based optimization, demonstrating minimal synchronization overhead in CC-NVM.
• Assise outperforms Orion by 69× and Ceph by 554× in throughput at scale, and breaks the MinuteSort world record by 1.5× due to its low-latency, high-throughput design.