[Paper Review] Systematic Generalization: What Is Required and Can It Be Learned?

Dzmitry Bahdanau, Shikhar Murty+4 more|arXiv (Cornell University)|Nov 30, 2018

Topic Modeling33 references25 citations

TL;DR

This paper investigates systematic generalization in visual question answering using a synthetic dataset (SQOOP) that tests models' ability to reason about all possible object pairs after training on only a subset. It finds that modular neural networks (NMNs) with hand-crafted, tree-structured layouts generalize significantly better than generic models or end-to-end NMNs, which often learn non-compositional, suboptimal layouts that hurt generalization—highlighting the need for explicit inductive biases or regularizers to achieve robust systematic reasoning.

ABSTRACT

Numerous models for grounded language understanding have been recently proposed, including (i) generic models that can be easily adapted to any given task and (ii) intuitively appealing modular models that require background knowledge to be instantiated. We compare both types of models in how much they lend themselves to a particular form of systematic generalization. Using a synthetic VQA test, we evaluate which models are capable of reasoning about all possible object pairs after training on only a small subset of them. Our findings show that the generalization of modular models is much more systematic and that it is highly sensitive to the module layout, i.e. to how exactly the modules are connected. We furthermore investigate if modular models that generalize well could be made more end-to-end by learning their layout and parametrization. We find that end-to-end methods from prior work often learn inappropriate layouts or parametrizations that do not facilitate systematic generalization. Our results suggest that, in addition to modularity, systematic generalization in language understanding may require explicit regularizers or priors.

Motivation & Objective

• To evaluate whether modular neural network architectures (NMNs) support stronger systematic generalization than generic neural models in visual question answering.
• To investigate how module layout and parametrization affect systematic generalization performance.
• To assess whether end-to-end learning of layout and parametrization in NMNs can preserve or improve systematic generalization.
• To identify whether existing end-to-end methods for NMNs converge to compositional, systematic solutions or suboptimal, non-compositional ones.
• To determine whether explicit regularizers or priors are necessary to guide learning toward systematic, compositional reasoning in neural models.

Proposed method

• The authors introduce SQOOP, a synthetic VQA dataset where models must answer spatial relational questions (e.g., 'Is there a letter A left of a digit 5?') about randomly paired objects in images.
• Models are trained on a small subset of object pairs but evaluated on all possible pairs to test systematic generalization.
• The study compares generic models (e.g., FiLM, MAC, RelNet) with modular NMNs using hand-crafted modules and fixed layouts.
• End-to-end variants of NMNs are evaluated, including layout induction (via learned parsers) and parametrization learning via soft-attention over questions.
• The performance of models is analyzed across different layout structures (e.g., tree vs. chain) and training signal strengths.
• Experiments are conducted on multiple SQOOP splits with increasing object pair diversity (e.g., #rhs/lhs=1 to #rhs/lhs=18), measuring zero-shot generalization to unseen pairs.

Experimental results

Research questions

• RQ1 Can modular neural network architectures (NMNs) achieve stronger systematic generalization than generic neural models in visual question answering?
• RQ2 How does the structural layout of modules (e.g., tree vs. chain) affect systematic generalization performance?
• RQ3 Do end-to-end methods that learn module layout or parametrization from data preserve systematic generalization, or do they converge to non-compositional solutions?
• RQ4 Is the performance of end-to-end NMNs sensitive to initialization, especially in high-complexity settings with many unseen object pairs?
• RQ5 What role do inductive biases or explicit regularizers play in enabling systematic generalization in neural models?

Key findings

• Modular NMNs with hand-crafted, tree-structured layouts generalize significantly better than generic models like FiLM, MAC, and RelNet, especially on unseen object pairs.
• The performance of NMNs is highly sensitive to layout: tree-structured layouts generalize much more strongly than chain-structured layouts, particularly on the hardest split (#rhs/lhs=18).
• End-to-end NMNs that learn layout or parametrization often fail to converge to tree-like, compositional structures, instead learning non-compositional chains or blurred attention mechanisms.
• Even with strong supervision, layout induction methods show high sensitivity to initialization and often fail to learn systematic solutions, indicating a need for explicit inductive biases.
• Parametrization induction shows promise on simpler splits (#rhs/lhs=2), suggesting that a richer training signal or prior may be sufficient to guide end-to-end NMNs toward systematic behavior.
• The results challenge the assumption that end-to-end learning alone is sufficient for systematic generalization, implying that explicit regularizers or architectural priors are necessary to achieve robust compositional reasoning.