[Paper Review] Learning to Compose Neural Networks for Question Answering

Jacob Andreas, Marcus Rohrbach+2 more|arXiv (Cornell University)|Jan 7, 2016

Multimodal Machine Learning ApplicationsComputer Science121 citations

TL;DR

A dynamic neural module network automatically assembles differentiable modules into question-specific architectures to answer questions about images and structured knowledge bases, trained with reinforcement learning using only (world, question, answer) triples.

ABSTRACT

We describe a question answering model that applies to both images and structured knowledge bases. The model uses natural language strings to automatically assemble neural networks from a collection of composable modules. Parameters for these modules are learned jointly with network-assembly parameters via reinforcement learning, with only (world, question, answer) triples as supervision. Our approach, which we term a dynamic neural model network, achieves state-of-the-art results on benchmark datasets in both visual and structured domains.

Motivation & Objective

• Enable question answering over both visual and structured world representations using modular neural components.
• Automatically assemble network layouts from questions without layout supervision.
• Train modules and layout predictor jointly to maximize answer accuracy.
• Leverage continuous representations to bridge perception and structured reasoning.

Proposed method

• Define a library of differentiable neural modules (e.g., find, lookup, relate, describe, exists).
• Represent questions as layouts z that specify how to compose modules into a neural network (JzKw) to process world representations w.
• Train modules with automatic supervision via backpropagation when z is fixed; use REINFORCE to learn layout selection p(z|x; θℓ) without layout annotations.
• Score candidate layouts generated from a dependency parse to a small candidate set; select layouts by a neural scoring model and optimize with policy gradient.
• Execute the chosen layout to obtain p(y|z,w; θe); update θe via standard backpropagation and θℓ via the REINFORCE gradient with reward based on answer accuracy.
• Extend modules to non-visual worlds by introducing relate and exists modules that operate on attention over entities or knowledge base elements.

Experimental results

Research questions

• RQ1 Can a system learn to compose neural networks from a question to answer across varying domains (images and structured data)?
• RQ2 Does joint learning of module parameters and dynamic network layouts improve QA performance without supervision on layouts?
• RQ3 Do continuous, differentiable modules enable effective reasoning over structured world representations as well as visual data?
• RQ4 Is reinforcement learning a viable approach to train layout prediction using only (world, question, answer) triples?

Key findings

• Achieved state-of-the-art results on visual question answering (VQA) and a compositional geography QA task (GeoQA) using dynamic assembly of neural modules.
• Outperformed prior models on VQA with a dynamic network that selects simple layouts (describe and and find) and, on GeoQA, achieved higher accuracy than both a purely logical model and a fixed-structure NMN.
• Dynamic networks provided gains particularly on questions requiring compositional reasoning and quantification.
• The model learns to focus attention on relevant regions in images and on relevant entities in knowledge bases, enabling accurate answers in many cases.
• In GeoQA+Q (with quantifiers), the dynamic layout predictor yielded notable improvements over fixed-structure baselines.