[Paper Review] Improving Systematic Generalization Through Modularity and Augmentation

Laura Ruis, Brenden M. Lake|arXiv (Cornell University)|Feb 22, 2022

Natural Language Processing Techniques4 citations

TL;DR

This paper improves systematic generalization in neural networks for grounded language understanding by combining modular architecture with structured data augmentation. It shows that modularity enables models to generalize to novel adverb-verb combinations—achieving 70% and 40% exact match gains on gSCAN benchmarks—while performance depends critically on the quality, not quantity, of augmented data, especially similarity to target adverbs.

ABSTRACT

Systematic generalization is the ability to combine known parts into novel meaning; an important aspect of efficient human learning, but a weakness of neural network learning. In this work, we investigate how two well-known modeling principles -- modularity and data augmentation -- affect systematic generalization of neural networks in grounded language learning. We analyze how large the vocabulary needs to be to achieve systematic generalization and how similar the augmented data needs to be to the problem at hand. Our findings show that even in the controlled setting of a synthetic benchmark, achieving systematic generalization remains very difficult. After training on an augmented dataset with almost forty times more adverbs than the original problem, a non-modular baseline is not able to systematically generalize to a novel combination of a known verb and adverb. When separating the task into cognitive processes like perception and navigation, a modular neural network is able to utilize the augmented data and generalize more systematically, achieving 70% and 40% exact match increase over state-of-the-art on two gSCAN tests that have not previously been improved. We hope that this work gives insight into the drivers of systematic generalization, and what we still need to improve for neural networks to learn more like humans do.

Motivation & Objective

• Address the challenge of systematic generalization in neural networks, particularly for adverb-verb compositionality that transforms action sequences.
• Investigate whether modularity and data augmentation can improve generalization to unseen combinations of known linguistic elements in grounded language tasks.
• Determine the role of vocabulary size and adverb similarity in enabling transfer learning for novel adverb-verb pairs.
• Evaluate how structured data augmentation, based on inferred rules of adverb behavior, affects model performance on systematic generalization tasks.
• Provide insights into the inductive biases and data quality factors that enable neural networks to generalize more like humans in compositional language understanding.

Proposed method

• Decompose the language understanding task into modular components—perception, navigation, and reasoning—using a modular neural network architecture.
• Design a structured data augmentation technique that infers rules governing adverb behavior (e.g., 'cautiously' requires a specific turning pattern) to generate synthetic training examples.
• Generate augmented datasets with up to 40× more examples than the original, focusing on adverb types with similar behavioral effects on action sequences.
• Train non-modular and modular models on varying subsets of augmented data, including different adverb types and similarity levels.
• Evaluate performance on gSCAN benchmarks, particularly on few-shot generalization to unseen adverb-verb combinations like 'pull while spinning' and 'cycle cautiously'.
• Use ablation studies to isolate the impact of vocabulary size, adverb similarity, and data distribution on generalization performance.

Experimental results

Research questions

• RQ1 Can modularity in neural network architecture improve systematic generalization for adverb-verb compositionality in grounded language tasks?
• RQ2 Does structured data augmentation—based on behavioral rules of adverbs—enable better generalization than naive data expansion?
• RQ3 How does the similarity between augmented adverbs and target test adverbs affect transfer performance?
• RQ4 Is the quantity of training examples sufficient for strong generalization, or is data quality (in terms of behavioral similarity) more critical?
• RQ5 To what extent can a model generalize to novel combinations when trained on diverse but behaviorally dissimilar adverbs?

Key findings

• A modular neural network architecture achieved 70% and 40% exact match improvements over state-of-the-art on two gSCAN tests involving novel adverb-verb combinations.
• Training on a non-modular model with nearly 40× more examples (including diverse adverbs) failed to achieve systematic generalization to 'cycle cautiously' or 'pull while spinning'.
• Performance on the 'cautiously' test split improved dramatically when a single adverb with similar behavioral rules (e.g., same turning pattern) was added, suggesting nearest-neighbor-style reasoning.
• Adding adverbs from a different behavioral class (e.g., 'while spinning' instead of 'cautiously') hurt performance on the 'cautiously' split, indicating that mismatched data can degrade generalization.
• The number of examples of the target adverb type (e.g., 'cautiously') had diminishing returns after 10 examples, indicating that data quantity alone is insufficient.
• The quality of augmented data—specifically, behavioral similarity to the target adverb—was more important than sheer volume for enabling systematic generalization.