[Paper Review] Multi-Mention Learning for Reading Comprehension with Neural Cascades

Swabha Swayamdipta, Ankur P. Parikh+1 more|arXiv (Cornell University)|Nov 2, 2017

Topic Modeling23 references3 citations

TL;DR

This paper proposes a cascaded neural network architecture for reading comprehension that uses lightweight feed-forward networks with attention to aggregate multiple mentions of answer candidates across long documents. By processing evidence in a hierarchical cascade, the model achieves state-of-the-art performance on TriviaQA, outperforming complex recurrent models while being significantly faster on long contexts, especially at scale beyond 10,000 tokens.

ABSTRACT

Reading comprehension is a challenging task, especially when executed across longer or across multiple evidence documents, where the answer is likely to reoccur. Existing neural architectures typically do not scale to the entire evidence, and hence, resort to selecting a single passage in the document (either via truncation or other means), and carefully searching for the answer within that passage. However, in some cases, this strategy can be suboptimal, since by focusing on a specific passage, it becomes difficult to leverage multiple mentions of the same answer throughout the document. In this work, we take a different approach by constructing lightweight models that are combined in a cascade to find the answer. Each submodel consists only of feed-forward networks equipped with an attention mechanism, making it trivially parallelizable. We show that our approach can scale to approximately an order of magnitude larger evidence documents and can aggregate information at the representation level from multiple mentions of each answer candidate across the document. Empirically, our approach achieves state-of-the-art performance on both the Wikipedia and web domains of the TriviaQA dataset, outperforming more complex, recurrent architectures.

Motivation & Objective

• Address the limitation of existing neural models in handling long or multi-document evidence by aggregating information from multiple mentions of the same answer.
• Overcome the inefficiency and scalability issues of recurrent architectures like biLSTMs when applied to long documents in reading comprehension.
• Design a model that can effectively leverage co-reference and distributed evidence across entire Wikipedia or web pages, rather than relying on truncated passages.
• Improve performance on the TriviaQA benchmark by combining evidence from multiple spans and contexts, especially when the answer is not concentrated in a single segment.
• Enable faster inference on long documents by using parallelizable feed-forward networks instead of autoregressive RNNs, particularly benefiting long-context scenarios.

Proposed method

• Construct a three-level cascaded model: Level 1 uses bag-of-embeddings for question, candidate span, and local context; Level 2 applies an attention mechanism to align question words with the span’s context; Level 3 aggregates representations from multiple mentions of the same answer candidate.
• Use a multi-loss training framework (Al-Rfou et al., 2016) where all submodels are trained simultaneously with gradients flowing from higher to lower levels, preventing higher-level models from subsuming lower-level features.
• Employ feed-forward networks throughout, enabling full parallelization and efficient GPU utilization, especially beneficial for long documents.
• Represent each answer candidate using a combination of question embedding, span embedding, and context embedding, with attention mechanisms to focus on relevant words.
• At inference, only the top-level classifier (Level 3) produces final predictions, which are based on aggregated representations from multiple mentions.
• Train the model end-to-end with a joint objective that encourages each level to learn complementary features, improving robustness and generalization.

Experimental results

Research questions

• RQ1 Can a cascaded architecture of lightweight feed-forward networks with attention outperform complex recurrent models on long-context reading comprehension tasks?
• RQ2 Does aggregating multiple mentions of the same answer across a document improve performance on long-form evidence like Wikipedia or web pages?
• RQ3 Can a multi-loss training strategy prevent higher-level models from dominating lower-level features, preserving useful representations from earlier stages?
• RQ4 To what extent does the model benefit from context around the answer span, especially when the span is ambiguous or out of context?
• RQ5 How does the model’s inference speed scale with document length compared to standard biLSTM-based approaches?

Key findings

• The model achieves state-of-the-art performance on both the Wikipedia and web domains of the TriviaQA dataset, outperforming more complex recurrent architectures.
• On documents of length 10,000 tokens, the model is approximately 45× faster than a standard biLSTM due to better GPU parallelization, demonstrating superior scalability.
• Aggregating multiple mentions of the same answer across a document significantly improves performance, as shown in cases where the correct answer appears in multiple non-adjacent spans.
• The model successfully avoids selecting the most frequent entity (e.g., 'Doctor Who') when it is not the correct answer, showing that aggregation helps in resolving ambiguity.
• Lower-level models (Level 1 and 2) often predict the correct answer among top candidates, but only the aggregated Level 3 model consistently selects the correct answer, proving the value of multi-mention aggregation.
• Despite strong performance, the model still struggles with entity-type confusion (e.g., mistaking 'Doctor Who' for 'Davros'), indicating limitations in coreference and semantic disambiguation.