[Paper Review] Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks

Brenden M. Lake, Marco Baroni|arXiv (Cornell University)|Oct 30, 2017

Natural Language Processing TechniquesComputer Science204 citations

TL;DR

The paper introduces SCAN, a grounded navigation command dataset, and systematically evaluates various seq2seq RNN models on their ability to generalize compositionally, revealing strong zero-shot generalization in some cases but severe failures to exploit true systematic compositionality, with a brief translation experiment as a proof of concept.

ABSTRACT

Humans can understand and produce new utterances effortlessly, thanks to their compositional skills. Once a person learns the meaning of a new verb "dax," he or she can immediately understand the meaning of "dax twice" or "sing and dax." In this paper, we introduce the SCAN domain, consisting of a set of simple compositional navigation commands paired with the corresponding action sequences. We then test the zero-shot generalization capabilities of a variety of recurrent neural networks (RNNs) trained on SCAN with sequence-to-sequence methods. We find that RNNs can make successful zero-shot generalizations when the differences between training and test commands are small, so that they can apply "mix-and-match" strategies to solve the task. However, when generalization requires systematic compositional skills (as in the "dax" example above), RNNs fail spectacularly. We conclude with a proof-of-concept experiment in neural machine translation, suggesting that lack of systematicity might be partially responsible for neural networks' notorious training data thirst.

Motivation & Objective

• Introduce SCAN as a controlled, compositional command-to-action task grounded in navigation.
• Evaluate a wide range of seq2seq recurrent architectures on zero-shot, systematic generalization tasks.
• Identify where standard models succeed in generalization and where they fail to exhibit true compositionality.
• Explore implications for neural machine translation and broader sequence-to-sequence learning regarding data efficiency and systematicity.

Proposed method

• Treat SCAN as a supervised seq2seq semantic parsing problem with 13 input words and 6 output actions.
• Systematically test SRNs, LSTMs, and GRUs with and without attention across 1-2 layer configurations and various hidden sizes.
• Perform extensive hyperparameter search (180 architectures, 5 seeds each) to identify top performers.
• Train models on 100,000 trials with Adam optimization and dropout, using teacher forcing and both teacher-forcing and free-running decoding regimes.
• Evaluate zero-shot generalization on carefully designed splits that stress compositionality and sequence length beyond training exposure.
• Include a proof-of-concept MT experiment to probe whether systematicity gaps extend to other seq2seq domains.

Experimental results

Research questions

• RQ1 Can standard seq2seq RNNs generalize to novel commands built by mixing known parts (mix-and-match) in SCAN?
• RQ2 Do RNNs exhibit true systematic compositionality when training examples require generalizing to longer action sequences or new combinations of primitives and modifiers?
• RQ3 How does attention influence generalization for different architectures (SRN, LSTM, GRU) on the SCAN tasks?
• RQ4 To what extent does a lack of systematicity in seq2seq models impact broader seq2seq tasks such as machine translation when introducing new vocabulary?
• RQ5 What strategies could encourage neural models to learn abstract rules rather than surface pattern recognition?

Key findings

• RNNs generalize well when test commands are composed from familiar parts, achieving near-perfect accuracy in some random-subset splits.
• Most models fail to generalize when test commands require systematic composition beyond observed patterns, with significant drops as action sequence length increases (best 13.8% average in long-sequence experiment).
• Turn-left generalization is strong for many models, but jump generalization is poor unless training exposes related composed forms, highlighting uneven systematicity.
• In long-sequence generalization, even with oracle length guidance, performance remains far from perfect, and accuracy strongly declines with longer action sequences (e.g., 95.76% for 24 actions vs 22.8% for 48 actions).
• Providing additional composed examples for a primitive like "jump" improves generalization substantially (e.g., 38.3% with 8, 77.8% with 16, 88.4% with 32 composed jump commands).
• A brief MT experiment shows that adding a new word (daxy) drastically degrades translation unless extensive exposure to composed uses is provided, suggesting broader limits to compositional generalization in seq2seq models.