[Paper Review] Learning Memory Access Patterns

Milad Hashemi, Kevin Swersky+6 more|arXiv (Cornell University)|Mar 6, 2018

Parallel Computing and Optimization Techniques33 references67 citations

TL;DR

This paper explores LSTM-based neural prefetchers for memory accesses, framing prefetching as a classification task over deltas (and PCs), and shows improved precision and recall over traditional hardware prefetchers on diverse benchmarks.

ABSTRACT

The explosion in workload complexity and the recent slow-down in Moore's law scaling call for new approaches towards efficient computing. Researchers are now beginning to use recent advances in machine learning in software optimizations, augmenting or replacing traditional heuristics and data structures. However, the space of machine learning for computer hardware architecture is only lightly explored. In this paper, we demonstrate the potential of deep learning to address the von Neumann bottleneck of memory performance. We focus on the critical problem of learning memory access patterns, with the goal of constructing accurate and efficient memory prefetchers. We relate contemporary prefetching strategies to n-gram models in natural language processing, and show how recurrent neural networks can serve as a drop-in replacement. On a suite of challenging benchmark datasets, we find that neural networks consistently demonstrate superior performance in terms of precision and recall. This work represents the first step towards practical neural-network based prefetching, and opens a wide range of exciting directions for machine learning in computer architecture research.

Motivation & Objective

• Motivate the memory wall challenge and the need for scalable predictive techniques beyond table-based hardware predictors.
• Investigate whether sequence models, specifically LSTMs, can predict memory misses to guide prefetching.
• Develop two LSTM-based prefetcher architectures and assess their practicality and accuracy.
• Evaluate neural prefetchers against traditional hardware prefetchers on realistic benchmark traces.
• Provide insights into the learned structure of memory access traces and potential directions for hardware integration.

Proposed method

• Formulate prefetching as a discrete classification problem by treating the address space as a vocabulary of delta values or clusters.
• Develop embedding LSTM with input PCs and deltas and a top-10 delta prediction output for prefetching.
• Develop clustering + LSTM to model local regions of address space with shared weights and cluster IDs as features.
• Train using offline traces from SPEC CPU2006 benchmarks and Google web search workloads, with 70/30 train/test split.
• Compare against a 10-stream hardware prefetcher and a GHB PC/DC correlation prefetcher, evaluating precision-at-10 and recall-at-10.
• Use deltas (Addr_{N+1}-Addr_N) as inputs to improve consistency and reduce vocabulary size within clusters.

Experimental results

Research questions

• RQ1 Can sequence-based models (LSTMs) outperform traditional hardware prefetchers in precision and recall for memory accesses?
• RQ2 Does discretizing the output space (via deltas or clustered addresses) enable effective neural prefetching at realistic hardware scales?
• RQ3 How do input modalities ( PCs vs deltas ) contribute to predictive information for prefetching?
• RQ4 What are the trade-offs between an embedding-based LSTM and a clustering-based LSTM in terms of accuracy, model size, and locality of memory accesses?

Key findings

• Neural network prefetchers achieve higher precision and recall than traditional hardware prefetchers on multiple benchmarks.
• Embedding LSTM and clustering + LSTM perform comparably in precision, while clustering + LSTM yields higher recall due to multiple vocabularies.
• Using deltas as inputs provides most predictive information for precision, while PCs contribute to recall.
• Clustering the address space into regions reduces vocabulary size and model footprint, enabling multi-task LSTMs with shared weights.
• The models demonstrate interpretable structure in learned representations (e.g., t-SNE visualizations reveal code-pattern semantics).