[Paper Review] Compositional generalization through meta sequence-to-sequence learning

Brenden M. Lake|arXiv (Cornell University)|Jun 12, 2019

Topic Modeling34 references44 citations

TL;DR

The paper introduces meta sequence-to-sequence (meta seq2seq) learning with memory-augmented networks to achieve compositional generalization on SCAN-like tasks, outperforming standard seq2seq and approaching human-like generalization in several experiments. It demonstrates memory-based abstraction of variables and rapid learning of new primitives through episodic meta-training, while noting limits on extrapolation to longer sequences.

ABSTRACT

People can learn a new concept and use it compositionally, understanding how to "blicket twice" after learning how to "blicket." In contrast, powerful sequence-to-sequence (seq2seq) neural networks fail such tests of compositionality, especially when composing new concepts together with existing concepts. In this paper, I show how memory-augmented neural networks can be trained to generalize compositionally through meta seq2seq learning. In this approach, models train on a series of seq2seq problems to acquire the compositional skills needed to solve new seq2seq problems. Meta se2seq learning solves several of the SCAN tests for compositional learning and can learn to apply implicit rules to variables.

Motivation & Objective

• Motivate and quantify the limitations of standard seq2seq models in compositional generalization.
• Propose a memory-augmented meta seq2seq learner trained over episodic tasks to learn how to learn compositionally.
• Demonstrate the approach on SCAN-derived tasks, including mutual exclusivity, adding new primitives, combining familiar concepts, and longer sequence generalization.

Proposed method

• Use a memory-augmented seq2seq backbone where an external key-value memory stores support items from each episode.
• Encode queries and support items with bidirectional LSTMs; store final support encodings as memory keys and values.
• Compute stepwise context by attending over memory to produce a per-step context for the decoder.
• Train across episodes (meta-training) where each episode provides a novel seq2seq problem with support and query items; update weights once per episode (no further weight updates during testing).
• In ablations, compare against standard seq2seq without memory, and test the impact of removing support loss or decoder attention.

Experimental results

Research questions

• RQ1 Can a memory-augmented meta-learning framework enable systematic compositional generalization in seq2seq tasks?
• RQ2 To what extent can the model learn to apply implicit rules to variables and generalize to novel compositions and longer sequences?
• RQ3 Do additional training aids (support loss, decoder attention) materially impact performance and memory utilization?
• RQ4 How does meta seq2seq perform on extended, augmented primitive sets and in tasks requiring rapid acquisition of new primitives?
• RQ5 What are the limits of meta seq2seq in extrapolating to longer output sequences?

Key findings

• Meta seq2seq achieves strong compositional generalization on SCAN-style tasks, outperforming standard seq2seq and syntactic attention baselines.
• In the mutual exclusivity task, the model achieves 100% accuracy by leveraging external memory to map unseen symbols based on supporting items.
• On the original SCAN add-jump task, meta seq2seq reaches around 99% accuracy, while standard seq2seq fails near 0%.
• With augmentation (larger primitive sets), meta seq2seq attains about 98.7% accuracy on test using four original primitives, whereas standard seq2seq remains around 12% and syntactic attention around 29% (variability across runs).
• Meta seq2seq succeeds in inferring and composing a new primitive (e.g., jump) with high accuracy under meta-training, but performance drops when test demands longer sequences or extrapolation beyond trained lengths, indicating limits in truly systematic generalization.