[Paper Review] Bandana: Using Non-volatile Memory for Storing Deep Learning Models

Assaf Eisenman, Maxim Naumov+6 more|arXiv (Cornell University)|Nov 14, 2018

Caching and Content Delivery20 references41 citations

TL;DR

Bandana uses non-volatile memory (NVM) with a small DRAM cache to store embeddings for recommender systems, employing hypergraph-based partitioning and miniature-cache simulations to boost NVM read bandwidth by 2–3×.

ABSTRACT

Typical large-scale recommender systems use deep learning models that are stored on a large amount of DRAM. These models often rely on embeddings, which consume most of the required memory. We present Bandana, a storage system that reduces the DRAM footprint of embeddings, by using Non-volatile Memory (NVM) as the primary storage medium, with a small amount of DRAM as cache. The main challenge in storing embeddings on NVM is its limited read bandwidth compared to DRAM. Bandana uses two primary techniques to address this limitation: first, it stores embedding vectors that are likely to be read together in the same physical location, using hypergraph partitioning, and second, it decides the number of embedding vectors to cache in DRAM by simulating dozens of small caches. These techniques allow Bandana to increase the effective read bandwidth of NVM by 2-3x and thereby significantly reduce the total cost of ownership.

Motivation & Objective

• Reduce DRAM usage for embedding tables in large-scale recommender systems.
• Explore NVM as primary storage for embeddings to lower total cost of ownership (TCO).
• Increase effective NVM read bandwidth by organizing data and caching intelligently.
• Evaluate partitioning and caching strategies to determine practical performance gains.

Proposed method

• Evaluate NVM read bandwidth characteristics and block granularity (4 KB) for embedding reads.
• Apply two embedding-vector partitioning strategies: supervised SHP hypergraph partitioning and unsupervised K-means clustering.
• Use LRU-based caching with a per-table threshold t to decide which vectors to cache after prefetching.
• Introduce miniature caches to simulate multiple cache thresholds and select the best one in real time.
• Incorporate a shadow cache to admit prefetched vectors and manage their eviction impact.

Experimental results

Research questions

• RQ1 Can NVM store large embedding tables with acceptable latency and throughput for real-time recommendation workloads?
• RQ2 How can embedding vectors be partitioned to maximize co-access and thus NVM bandwidth?
• RQ3 What caching and admission-control strategies best increase the effective bandwidth from NVM for embeddings?
• RQ4 Can miniature-cache simulations reliably configure per-table cache thresholds to optimize performance?

Key findings

• Bandana increases the effective read bandwidth of NVM by approximately 2–3× compared to naive DRAM substitution.
• Hypergraph-based supervised partitioning (SHP) generally yields higher bandwidth gains than unsupervised K-means for the studied workloads.
• Prefetch-driven caching requires careful management; naive caching of all prefetched vectors can hurt performance due to evictions.
• Miniature caches enable near-ideal threshold selection with low overhead, achieving bandwidth gains close to oracle policies.
• The effectiveness of Bandana varies across embedding tables depending on access patterns; some tables show large bandwidth gains, others less so.
• Using SHP with an unlimited DRAM cache demonstrates the potential of NVM as a primary storage medium for embeddings.