[Paper Review] Self-Assembling Modular Networks for Interpretable Multi-Hop Reasoning

Yichen Jiang, Mohit Bansal|arXiv (Cornell University)|Sep 12, 2019

Topic Modeling32 references4 citations

TL;DR

This paper proposes a self-assembling neural modular network (NMN) with dynamic controller-based module composition for interpretable multi-hop question answering on HotpotQA. The model uses Find, Relocate, Compare, and NoOp modules, guided by a controller RNN that predicts soft layouts, achieving state-of-the-art performance with strong interpretability through sub-question decomposition, expert-layout alignment, and intermediate bridge-entity inference.

ABSTRACT

Multi-hop QA requires a model to connect multiple pieces of evidence scattered in a long context to answer the question. The recently proposed HotpotQA (Yang et al., 2018) dataset is comprised of questions embodying four different multi-hop reasoning paradigms (two bridge entity setups, checking multiple properties, and comparing two entities), making it challenging for a single neural network to handle all four. In this work, we present an interpretable, controller-based Self-Assembling Neural Modular Network (Hu et al., 2017, 2018) for multi-hop reasoning, where we design four novel modules (Find, Relocate, Compare, NoOp) to perform unique types of language reasoning. Based on a question, our layout controller RNN dynamically infers a series of reasoning modules to construct the entire network. Empirically, we show that our dynamic, multi-hop modular network achieves significant improvements over the static, single-hop baseline (on both regular and adversarial evaluation). We further demonstrate the interpretability of our model via three analyses. First, the controller can softly decompose the multi-hop question into multiple single-hop sub-questions to promote compositional reasoning behavior of the main network. Second, the controller can predict layouts that conform to the layouts designed by human experts. Finally, the intermediate module can infer the entity that connects two distantly-located supporting facts by addressing the sub-question from the controller.

Motivation & Objective

• To address the challenge of multi-hop question answering where evidence is scattered across long contexts and requires multiple reasoning steps.
• To design a modular neural network that can dynamically assemble different reasoning modules based on the question's reasoning type.
• To improve interpretability by enabling the model to decompose questions into sub-questions and predict reasoning layouts similar to human-designed plans.
• To achieve strong performance on HotpotQA while maintaining transparency in reasoning steps through controller-based module selection and intermediate reasoning tracking.

Proposed method

• A controller RNN generates a soft layout at each reasoning step by predicting a distribution over four modules: Find, Relocate, Compare, and NoOp.
• The Find module performs standard span extraction via bi-attention between question and context embeddings.
• The Relocate module conditions on a previous answer to shift attention to a new sub-question, enabling sequential reasoning.
• The Compare module compares outputs from two prior modules to answer yes/no questions, using question cues like 'same' or 'different'.
• The NoOp module preserves the current state when no further reasoning is needed.
• The model uses differentiable stack-based memory and continuous layout prediction to enable end-to-end training via gradient descent.

Experimental results

Research questions

• RQ1 Can a neural modular network dynamically compose reasoning modules based on the question type to solve multi-hop QA tasks?
• RQ2 To what extent can the controller RNN decompose complex questions into meaningful sub-questions that reflect compositional reasoning?
• RQ3 How well does the predicted modular layout align with human-designed expert layouts for different reasoning paradigms?
• RQ4 Can the model infer intermediate bridge entities that connect distant supporting facts in a reasoning chain?
• RQ5 Does integrating BERT as a context encoder improve performance while preserving interpretability in the modular architecture?

Key findings

• The model achieves significant improvements over static single-hop baselines on both standard and adversarial evaluations on HotpotQA.
• The controller predicts layouts matching human expert designs in 99.9% of bridge-type questions and 68.6% of comparison-type questions with Yes/No answers.
• The controller successfully decomposes multi-hop questions into sub-questions, with attention maps aligning with the intended reasoning steps in 4 out of 4 analyzed examples.
• The model identifies intermediate bridge entities (e.g., 'Shirley Temple') correctly in 100% of bridge-type questions, demonstrating effective reasoning chain construction.
• When enhanced with BERT embeddings, the model achieves 56.63 EM and 71.26 F1, matching fine-tuned BERT-base performance while enabling superior interpretability.
• The BERT-enhanced NMN controller achieves 96.9% alignment with expert layouts for comparison-type questions, significantly outperforming the non-BERT version.