[Paper Review] FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension

Hsin-Yuan Huang, Chenguang Zhu+2 more|arXiv (Cornell University)|Nov 16, 2017

Topic Modeling86 citations

TL;DR

FusionNet proposes history-of-word based fully-aware attention and multi-level fusion to context and question, achieving state-of-the-art results on SQuAD and adversarial datasets at the time.

ABSTRACT

This paper introduces a new neural structure called FusionNet, which extends existing attention approaches from three perspectives. First, it puts forward a novel concept of "history of word" to characterize attention information from the lowest word-level embedding up to the highest semantic-level representation. Second, it introduces an improved attention scoring function that better utilizes the "history of word" concept. Third, it proposes a fully-aware multi-level attention mechanism to capture the complete information in one text (such as a question) and exploit it in its counterpart (such as context or passage) layer by layer. We apply FusionNet to the Stanford Question Answering Dataset (SQuAD) and it achieves the first position for both single and ensemble model on the official SQuAD leaderboard at the time of writing (Oct. 4th, 2017). Meanwhile, we verify the generalization of FusionNet with two adversarial SQuAD datasets and it sets up the new state-of-the-art on both datasets: on AddSent, FusionNet increases the best F1 metric from 46.6% to 51.4%; on AddOneSent, FusionNet boosts the best F1 metric from 56.0% to 60.7%.

Motivation & Objective

• Motivate improved machine reading comprehension by leveraging information across all representation levels from word embeddings to high-level features.
• Develop a lightweight fully-aware attention mechanism that uses history-of-word to measure cross-text interactions.
• Propose and validate a fully-aware multi-level fusion architecture that exploits question-context information layer by layer.
• Demonstrate state-of-the-art performance on SQuAD and robustness on adversarial datasets AddSent and AddOneSent.

Proposed method

• Introduce history-of-word (HoW) as the concatenation of all representations for a word across layers.
• Define a symmetric, nonlinearity-enhanced attention scoring function S(HoW_Ai, HoW_Bj) using a diagonal D in the decomposition U^T D U to stabilize training.
• Implement Fully-Aware Multi-Level Fusion: fuse word-level and higher-level features from Q to C with separate attentions; fuse C to Q at multiple levels; apply self-boosted fusion on C using HoW.
• Use a BiLSTM-based reading and multi-layer fusion to produce context and question understanding vectors U_C and U_Q, then predict span start and end in SQuAD via attention over these vectors.
• Evaluate on SQuAD and adversarial SQuAD datasets (AddSent, AddOneSent) and compare with contemporary models.

Experimental results

Research questions

• RQ1 Can history-of-word captured across all representation levels improve attention and fusion in MRC?
• RQ2 Does a symmetric, nonlinearity-enhanced fully-aware attention provide training stability and performance gains?
• RQ3 Does multi-level, fully-aware fusion outperform single-level or naive fusion strategies in MRC?
• RQ4 How does FusionNet perform on standard and adversarial MRC benchmarks?

Key findings

• FusionNet achieves top results on SQuAD in both single and ensemble configurations at the time of writing (EM 78.8%, F1 85.9% for ensemble).
• On adversarial datasets AddSent and AddOneSent, FusionNet increases best F1 from 46.6% to 51.4% and from 56.0% to 60.7%, respectively.
• Fully-aware multi-level fusion with history-of-word consistently improves EM/F1 over non-fully-aware or single-level approaches.
• Symmetric attention with nonlinearity outperforms additive, multiplicative, and scaled variants in ablation studies.
• The model demonstrates strong generalization to adversarial inputs, indicating improved language understanding capabilities.