[Paper Review] Associative Long Short-Term Memory

Ivo Danihelka, Greg Wayne+3 more|arXiv (Cornell University)|Feb 9, 2016

Advanced Memory and Neural ComputingEngineering18 references108 citations

TL;DR

The paper introduces Associative LSTM, a memory-augmented RNN that uses redundant holographic memory to store key–value pairs without increasing network parameters, improving memorization speed and capacity. It integrates HRR-based binding with LSTM gates and multiple read/write copies to reduce retrieval noise.

ABSTRACT

We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.

Motivation & Objective

• Motivate enhancing LSTM with memory mechanisms without adding parameters.
• Develop an associative, key–value memory using holographic reduced representations.
• Introduce redundancy through multiple memory copies to reduce retrieval noise.
• Integrate the redundant associative memory with LSTM gates to form a unified architecture.
• Demonstrate faster learning and competitive performance on memorization and sequence tasks.

Proposed method

• Represent key–value pairs using holographic reduced representations with binding via complex-valued operations.
• Create redundant memory traces by storing multiple transformed copies of each key–value pair with independent permutations.
• Retrieve by averaging across copies and using conjugate-like operations to bind/unbind keys.
• Integrate the associative memory into LSTM by producing complex-valued keys and update rules that mirror LSTM gating (forget, input, output) with complex-valued components.
• Allow parallel updating of copies and allow head-like reading via multiple keys (memory heads).
• Compare against baselines (LSTM, Permutation RNN, Unitary RNN, Multiplicative Unitary RNN) and assess learning speed and accuracy across tasks.

Experimental results

Research questions

• RQ1 Can an associative, memory-augmented LSTM store and retrieve key–value pairs with higher capacity without increasing parameter count?
• RQ2 Does redundant storage via multiple copies reduce retrieval noise and improve learning speed on memorization and sequence tasks?
• RQ3 How does associative memory integrate with LSTM gates to preserve sequence modeling capabilities while enabling memory addressing?
• RQ4 How does the Associative LSTM compare to standard LSTM and other memory-augmented models on canonical tasks (episodic copy, XML modeling, variable assignment, arithmetic, Wikipedia)?

Key findings

• The redundant associative memory enables larger effective memory capacity without increasing network parameters.
• Retrieval noise decreases as the number of stored items grows when using multiple copies with random permutations, approximately keeping retrieval error in check when copies scale with items.
• Associative LSTM achieves faster learning on episodic copy and XML modeling tasks compared with LSTM and competitive results on other tasks, especially as the number of copies increases.
• On the episodic copy task, associative memory with multiple copies improves speed; single-copy associative LSTM is competitive but not always superior to larger LSTM.
• On the XML modeling task, Associative LSTM shows significant advantages with more copies, outperforming or matching LSTM in several configurations.
• On variable assignment and arithmetic tasks, multiple reading/writing heads (copies) help Associative LSTM solve tasks more efficiently, though task details vary with copy count and architecture.
• On Wikipedia language modeling, Associative LSTM performs comparably to LSTM, indicating it is at least as general as LSTM for sequence modeling.