[Paper Review] A neural network memory prefetcher using semantic locality

Leeor Peled, Uri Weiser+1 more|arXiv (Cornell University)|Mar 19, 2018

Parallel Computing and Optimization Techniques47 references6 citations

TL;DR

This paper proposes a neural network-based memory prefetcher that leverages semantic locality by learning correlations between program context and memory access patterns. By training on runtime machine and program state, the prefetcher predicts future addresses with 30% average speedup on SPEC2006 and up to 4.6× on kernels, outperforming state-of-the-art heuristic prefetchers through superior associative learning and pattern detection.

ABSTRACT

Accurate memory prefetching is paramount for processor performance, and modern processors employ various techniques to identify and prefetch different memory access patterns. While most modern prefetchers target spatio-temporal patterns by matching memory addresses that are accessed in close proximity (either in space or time), the recently proposed concept of semantic locality views locality as an artifact of the algorithmic level and searches for correlations between memory accesses and program state. While this approach was shown to be effective, capturing semantic locality requires significant associative learning capabilities. In this paper we utilize neural networks for this task. We leverage recent advances in machine learning to propose a neural network prefetcher. We show that by observing program context, this prefetcher can learn distinct memory access patterns that cannot be covered by other state-of-the-art prefetchers. We evaluate the neural network prefetcher over SPEC2006, Graph500, and several microbenchmarks. We show that the prefetcher can deliver an average speedup of 30% for SPEC2006 (up to 2.7x) and up to 4.6x over kernels. We also present a high-level design of our prefetcher, explore the power, energy and area limitations, and propose several optimizations for feasibility. We believe that this line of research can further improve the efficiency of such neural networks and allow harnessing them for additional micro-architectural predictions.

Motivation & Objective

• To explore whether neural networks can outperform heuristic and reinforcement learning-based prefetchers in capturing complex memory access patterns.
• To investigate the feasibility of using neural networks for dynamic, context-aware memory prefetching based on semantic locality.
• To evaluate the performance, energy, and area trade-offs of implementing a neural network prefetcher in modern processors.
• To identify optimal network architectures and quantization strategies for efficient and accurate address prediction.
• To demonstrate that neural networks can unify multiple spatio-temporal prefetching heuristics into a single, adaptive learning engine.

Proposed method

• The prefetcher uses a feedforward neural network trained at runtime on program and machine context features to predict future memory accesses.
• Input features include program counter history, register states, and memory access patterns, encoded as vectors for network processing.
• The network is trained using backpropagation to minimize prediction error between predicted and actual future addresses.
• A high-level microarchitectural design is proposed, including optimizations for area, power, and energy efficiency.
• Quantization techniques (e.g., 3-bit weights) are evaluated to reduce precision and improve hardware feasibility.
• The system is evaluated using gem5 simulations across SPEC2006, Graph500, and hand-written kernels with dynamic training.

Experimental results

Research questions

• RQ1 Can a neural network effectively learn and generalize complex memory access patterns that are not captured by traditional spatio-temporal prefetchers?
• RQ2 How does the performance of a neural network prefetcher compare to state-of-the-art heuristic and reinforcement learning-based prefetchers in diverse workloads?
• RQ3 What are the trade-offs between network size, depth, precision, and prediction accuracy in a hardware-implementable prefetcher?
• RQ4 How does quantization affect the learning convergence and prediction accuracy of the neural network prefetcher?
• RQ5 In which workloads does semantic locality, as captured by neural networks, provide a significant advantage over context-RL or heuristic methods?

Key findings

• The neural network prefetcher achieves an average speedup of 30% on the SPEC2006 suite, with peak speedup reaching 2.7× on individual benchmarks.
• On selected kernels, the prefetcher delivers up to 4.6× speedup, significantly outperforming baseline and other state-of-the-art prefetchers.
• The neural network outperforms GHB-PC/DC by 43%, SMS by 2×, and VLDP by 2.9× in terms of performance gain on SPEC2006.
• Quantization to 3-bit precision causes minimal performance degradation in most benchmarks, though MCF and gemsFDTD show significant slowdown due to convergence issues.
• The neural network excels in workloads with complex, evolving data structures (e.g., LBM, prim, matmul) where context-RL struggles due to frequent state changes.
• The prefetcher demonstrates superior scalability in large data-set scenarios due to distributed weight storage, outperforming context-RL on longer skips and complex traversals.