[Paper Review] Learning to Transduce with Unbounded Memory

Edward Grefenstette, Karl Moritz Hermann+2 more|arXiv (Cornell University)|Jun 8, 2015

Natural Language Processing Techniques18 references138 citations

TL;DR

This paper proposes differentiable neural stacks, queues, and deques as unbounded memory mechanisms for recurrent networks, enabling them to learn and generalize transduction algorithms beyond training sequence lengths. Unlike standard LSTMs, these memory-augmented models achieve perfect generalization to longer sequences and converge orders of magnitude faster, demonstrating superior inductive bias for sequence-to-sequence tasks like copying, reversal, and morphological inflection.

ABSTRACT

Recently, strong results have been demonstrated by Deep Recurrent Neural Networks on natural language transduction problems. In this paper we explore the representational power of these models using synthetic grammars designed to exhibit phenomena similar to those found in real transduction problems such as machine translation. These experiments lead us to propose new memory-based recurrent networks that implement continuously differentiable analogues of traditional data structures such as Stacks, Queues, and DeQues. We show that these architectures exhibit superior generalisation performance to Deep RNNs and are often able to learn the underlying generating algorithms in our transduction experiments.

Motivation & Objective

• To investigate whether recurrent networks with unbounded, differentiable memory structures can generalize better than standard deep LSTMs on synthetic transduction tasks.
• To design memory mechanisms that mimic classical data structures (stacks, queues, deques) but are continuously differentiable for end-to-end training.
• To evaluate whether such memory-augmented models can learn the underlying algorithmic rules of transduction tasks rather than memorizing training data.
• To compare the performance and generalization capabilities of memory-enhanced LSTMs against standard deep LSTM benchmarks across diverse linguistic transduction tasks.

Proposed method

• The neural stack uses continuous push and pop operations parameterized by real values in (0,1), allowing differentiable updates to a vector stack with dynamic size.
• The neural queue extends the stack by modifying the update rule to prioritize the oldest element, enabling FIFO behavior through a shift-invariant update mechanism.
• The neural dequee combines stack and queue semantics by allowing both push/pop at the front and back, using separate control gates for each end.
• The controller network (LSTM) dynamically controls memory operations, with gradients backpropagated through the memory dynamics via exact partial derivatives.
• The memory structures are fully decoupled from the controller, enabling analyzable backward dynamics and stable training.
• The models are trained end-to-end on synthetic transduction tasks using cross-entropy loss, with evaluation on generalization to longer sequences than seen during training.

Experimental results

Research questions

• RQ1 Can differentiable neural stacks, queues, and deques outperform standard deep LSTMs in learning and generalizing sequence transduction algorithms?
• RQ2 Do memory-augmented models learn the underlying algorithmic rules of transduction tasks rather than memorizing training data?
• RQ3 Can these models generalize perfectly to sequences twice as long as those in the training set?
• RQ4 How do convergence speeds and parameter efficiency compare between memory-augmented models and standard deep LSTMs?
• RQ5 To what extent do different memory structures (stack, queue, dequee) enable the controller to learn distinct transduction patterns?

Key findings

• The DeQue-LSTM model achieved 100% accuracy on all tasks, including sequence inversion, copying, and gender conjugation, with perfect generalization to sequences up to twice the training length.
• Neural stack and queue models outperformed deep LSTMs significantly, especially in tasks requiring hierarchical or sequential ordering, such as SVO-to-SOV transformation.
• Enhanced models converged to optimal performance orders of magnitude faster than standard LSTMs, with convergence occurring in fewer than 100 training steps on most tasks.
• While deep LSTMs failed to generalize beyond training sequence lengths, memory-augmented models consistently maintained 100% accuracy on longer test sequences, indicating procedural learning over memorization.
• The neural dequee demonstrated the ability to emulate both stack and queue behavior, allowing a single controller to solve multiple distinct transduction tasks by switching memory access patterns.
• In tasks like bigram flipping, all models—including the best deep LSTMs—struggled with the final two symbols, suggesting a shared difficulty in modeling symmetric, non-local dependencies.