[Paper Review] Using Fast Weights to Attend to the Recent Past

Jimmy Ba, Geoffrey E. Hinton+3 more|arXiv (Cornell University)|Oct 20, 2016

Neural Networks and Applications153 citations

TL;DR

The paper introduces fast weights as a fast, associative memory that attends to recent hidden states, enabling improved memory and attention in RNNs, with layer normalization improving stability and performance across tasks.

ABSTRACT

Until recently, research on artificial neural networks was largely restricted to systems with only two types of variable: Neural activities that represent the current or recent input and weights that learn to capture regularities among inputs, outputs and payoffs. There is no good reason for this restriction. Synapses have dynamics at many different time-scales and this suggests that artificial neural networks might benefit from variables that change slower than activities but much faster than the standard weights. These "fast weights" can be used to store temporary memories of the recent past and they provide a neurally plausible way of implementing the type of attention to the past that has recently proved very helpful in sequence-to-sequence models. By using fast weights we can avoid the need to store copies of neural activity patterns.

Motivation & Objective

• Motivate the need for a memory mechanism faster than slow synaptic weights to store recent history without storing full activation patterns.
• Propose a fast associative memory using outer-product updates that decays over time to store recent hidden states.
• Demonstrate that fast weights enable improved performance on associative retrieval, MNIST visual attention, facial expression recognition, and reinforcement learning with memory.
• Show how layer normalization stabilizes fast weight dynamics and improves learning.
• Compare fast weights to standard RNNs and LSTMs across multiple tasks to establish benefits and limitations.

Proposed method

• Define a fast weight matrix A(t) that decays with rate λ and is updated by the outer product of the current hidden state h(t): A(t) = λ A(t-1) + η h(t) h(t)^T.
• Compute the next hidden state h(t+1) via a two-step process: first a preliminary h0(t+1) = f(W h(t) + C x(t)), then an inner loop that iteratively refines h by incorporating A(t) h_s(t+1) over S steps.
• Incorporate layer normalization into the inner loop to stabilize dynamics: h_{s+1}(t+1) = f(LN[ W h(t) + C x(t) + A(t) h_s(t+1) ]).
• Use slow weights W and C learned by SGD, and a fast learning rule for A with η and decay λ; train with mini-batches using Adam.
• Compare fast weights to IRNN, standard LSTM, and associative LSTM across tasks like associative retrieval, MNIST with glimpses, facial expression recognition, and reinforcement learning with memory.

Experimental results

Research questions

• RQ1 Can a fast, synapse-like memory store recent past information more efficiently than traditional hidden-state memories?
• RQ2 Does layer normalization improve the stability and performance of fast weight memories in recurrent networks?
• RQ3 Do fast weights offer tangible accuracy or learning-speed benefits on associative retrieval, visual attention with glimpses, facial expression recognition, and memory-based reinforcement learning compared to standard RNNs and LSTMs?

Key findings

• Fast weights substantially improve associative retrieval when the recurrent layer size is small, outperforming LSTM variants and accelerating convergence.
• In MNIST with a multi-level visual attention setup, fast weights achieve competitive or superior accuracy, especially with limited hidden units, compared to IRNN, LSTM, and a ConvNet baseline.
• On facial expression recognition, the fast weights model often surpasses IRNN and LSTM baselines, though ConvNet can still outperform the multi-level attention model in some settings.
• In reinforcement learning with memory (Catch task), agents using fast weights learn faster than those using ReLU RNN or LSTM, with stronger gains on more memory-demanding variants.
• Layer normalization improves robustness of fast weight dynamics, allowing stable training across a wider range of learning rates and decay parameters.