[Paper Review] End-To-End Memory Networks

Sainbayar Sukhbaatar, Arthur Szlam+2 more|arXiv (Cornell University)|Mar 31, 2015

Topic Modeling25 references689 citations

TL;DR

This paper introduces End-to-End Memory Networks, a differentiable neural network with a recurrent attention mechanism over an external memory, enabling end-to-end training without supervision on supporting facts. The model uses multiple memory hops to improve performance on question answering and language modeling, achieving competitive results with fewer parameters than LSTMs and outperforming RNNs on benchmark datasets like Penn Treebank and Text8.

ABSTRACT

We introduce a neural network with a recurrent attention model over a possibly large external memory. The architecture is a form of Memory Network (Weston et al., 2015) but unlike the model in that work, it is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings. It can also be seen as an extension of RNNsearch to the case where multiple computational steps (hops) are performed per output symbol. The flexibility of the model allows us to apply it to tasks as diverse as (synthetic) question answering and to language modeling. For the former our approach is competitive with Memory Networks, but with less supervision. For the latter, on the Penn TreeBank and Text8 datasets our approach demonstrates comparable performance to RNNs and LSTMs. In both cases we show that the key concept of multiple computational hops yields improved results.

Motivation & Objective

• To develop a neural network architecture that supports multiple computational hops over an external memory for reasoning tasks.
• To enable end-to-end training of memory networks without requiring supervision on intermediate supporting facts, increasing applicability to real-world tasks.
• To improve performance on question answering and language modeling by leveraging multiple attention hops over memory.
• To demonstrate that multiple hops and joint optimization of memory representations significantly enhance model generalization and performance.
• To show that the model can be scaled effectively to large-vocabulary language modeling tasks with minimal architectural modifications.

Proposed method

• The model stores input sequences as continuous memory vectors using an embedding matrix A, with queries similarly embedded via matrix B.
• Attention weights are computed via a softmax over the dot product between the query embedding and each memory vector, producing a probability distribution over memory locations.
• The output is a weighted sum of output vectors c_i, where weights are the attention probabilities, enabling differentiable memory read operations.
• Multiple hops are implemented by recursively updating the query representation using the output of each hop, with residual connections (u^{k+1} = u^k + o^k).
• Weight tying strategies are applied—adjacent and layer-wise—to reduce parameters and improve training stability.
• The final prediction is generated via a softmax over a final weight matrix W applied to the final query-output combination, trained via cross-entropy loss.

Experimental results

Research questions

• RQ1 Can a memory network be trained end-to-end without supervision on intermediate reasoning steps?
• RQ2 How does the number of memory hops affect performance in question answering and language modeling?
• RQ3 Can a differentiable memory mechanism outperform standard RNNs and LSTMs on language modeling benchmarks?
• RQ4 Does the use of multiple hops enable better modeling of long-term dependencies and context in sequential tasks?
• RQ5 How do weight tying and parameter sharing affect the generalization and scalability of the model?

Key findings

• The model achieves a perplexity of 111 on the Penn Treebank dataset, outperforming RNN/SCRN (115) and approaching LSTM performance with 1.5x fewer parameters than comparable RNNs.
• On the Text8 dataset, the model achieves a perplexity of 147, outperforming LSTM (154) despite having only 1.5x the number of parameters of a standard RNN.
• Increasing the number of memory hops consistently improves performance, demonstrating the importance of multi-hop reasoning in the model.
• Visualization of attention weights shows that different hops specialize—some focus on recent words, others attend broadly across memory—indicating complementary roles.
• Unlike RNNs, the memory does not decay exponentially; instead, it maintains consistent activation across memory positions, which may explain the performance gains.
• Gradient clipping with L2 norm thresholding of 50 was crucial for stable training, especially in deeper models with multiple hops.