[Paper Review] High Performance Data Persistence in Non-Volatile Memory for Resilient High Performance Computing

Yingchao Huang, Kai Wu+1 more|arXiv (Cornell University)|Apr 30, 2017

Advanced Data Storage Technologies29 references3 citations

TL;DR

This paper proposes using non-volatile memory (NVM) as main memory to enable high-performance, frequent data persistence in HPC systems, eliminating costly data copying in traditional checkpointing. By introducing in-place versioning and optimized cache flushing, the approach achieves persistent checkpoints with only 4.4% average runtime overhead, significantly reducing recomputation and overcoming the resilience and recomputation dilemmas inherent in conventional checkpointing.

ABSTRACT

Resilience is a major design goal for HPC. Checkpoint is the most common method to enable resilient HPC. Checkpoint periodically saves critical data objects to non-volatile storage to enable data persistence. However, using checkpoint, we face dilemmas between resilience, recomputation and checkpoint cost. The reason that accounts for the dilemmas is the cost of data copying inherent in checkpoint. In this paper we explore how to build resilient HPC with non-volatile memory (NVM) as main memory and address the dilemmas. We introduce a variety of optimization techniques that leverage high performance and non-volatility of NVM to enable high performance data persistence for data objects in applications. With NVM we avoid data copying; we optimize cache flushing needed to ensure consistency between caches and NVM. We demonstrate that using NVM is feasible to establish data persistence frequently with small overhead (4.4% on average) to achieve highly resilient HPC and minimize recomputation.

Motivation & Objective

• Address the resilience dilemma in future HPC systems, where shorter mean time between failures (MTBF) necessitates more frequent checkpoints but increases runtime overhead.
• Overcome the recomputation dilemma, where frequent checkpoints reduce data loss but incur high overhead due to data copying.
• Leverage the non-volatility and high performance of NVM to eliminate the need for traditional data copying in checkpointing.
• Develop a software-only solution that avoids OS or hardware modifications while ensuring data consistency between caches and NVM.
• Minimize runtime overhead of persistent checkpointing to enable frequent, efficient data persistence for resilient HPC execution.

Proposed method

• Introduce in-place versioning, a technique that uses application-inherent write operations to create new data object versions directly in NVM without explicit data copying.
• Derive transformation rules to automatically apply in-place versioning to source code, abstracting low-level details from programmers.
• Implement proactive cache flushing to ensure consistency between CPU caches and NVM after version creation, flushing all blocks of the new version.
• Optimize cache flushing using SIMD-based non-temporal instructions (e.g., MOVDQU) to bypass caches and reduce data movement between cache and memory.
• Parallelize cache flushing operations to improve performance and reduce latency during checkpointing.
• Use an epoch-based model to manage persistent writes, treating the period from version creation to cache flush as a single epoch where concurrent persistent writes are allowed.

Experimental results

Research questions

• RQ1 Can NVM-based main memory eliminate the need for data copying in HPC checkpointing, thereby resolving the resilience and recomputation dilemmas?
• RQ2 How can in-place versioning be automatically applied to applications to create persistent data versions without explicit data movement?
• RQ3 What optimization techniques can minimize the performance overhead of cache flushing in NVM-based persistent checkpointing?
• RQ4 Can a software-only approach achieve low runtime overhead without requiring OS or hardware support while ensuring data consistency?
• RQ5 To what extent can frequent, low-overhead data persistence be achieved using NVM to minimize recomputation and improve HPC resilience?

Key findings

• The proposed in-place versioning technique eliminates data copying in checkpointing by leveraging application write operations to directly update data in NVM.
• With optimizations including SIMD-based non-temporal instructions and parallel cache flushing, the average runtime overhead of persistent checkpointing is reduced to 4.4%.
• Even under optimistic NVM performance assumptions, traditional NVM-based checkpointing still incurs up to 46% overhead due to data copying, highlighting the need for copy-avoidance techniques.
• The proactive cache flushing strategy ensures data consistency between caches and NVM, though it incurs cost due to lack of tracking for dirty blocks.
• The approach achieves high resilience by enabling frequent data persistence with minimal performance impact, significantly reducing recomputation after failures.
• The solution requires no OS or hardware modifications, making it practical for deployment in existing HPC environments.