[Paper Review] Bidirectional Attention Flow for Machine Comprehension

Min Joon Seo, Aniruddha Kembhavi+2 more|arXiv (Cornell University)|Nov 5, 2016

Topic Modeling1,291 citations

TL;DR

BiDAF introduces a multi-stage, memory-less, bi-directional attention flow network that models query–context interactions without early context summarization, achieving state-of-the-art results on SQuAD and CNN/DailyMail cloze.

ABSTRACT

Machine comprehension (MC), answering a query about a given context paragraph, requires modeling complex interactions between the context and the query. Recently, attention mechanisms have been successfully extended to MC. Typically these methods use attention to focus on a small portion of the context and summarize it with a fixed-size vector, couple attentions temporally, and/or often form a uni-directional attention. In this paper we introduce the Bi-Directional Attention Flow (BIDAF) network, a multi-stage hierarchical process that represents the context at different levels of granularity and uses bi-directional attention flow mechanism to obtain a query-aware context representation without early summarization. Our experimental evaluations show that our model achieves the state-of-the-art results in Stanford Question Answering Dataset (SQuAD) and CNN/DailyMail cloze test.

Motivation & Objective

• Motivate and improve machine comprehension by better modeling interactions between question and context.
• Avoid early summarization of context into fixed-size vectors to preserve information flow.
• Propose a memory-less, bi-directional attention mechanism that flows into a hierarchical architecture.
• Combine character, word, and contextual embeddings with a multi-layer modeling stack to predict answer spans.

Proposed method

• Use character-level CNNs and pre-trained word embeddings, followed by a two-layer Highway network.
• Apply a contextual bidirectional LSTM to encode context and query separately.
• Compute a Bi-Directional Attention Flow by building a shared similarity matrix between context and query.
• Flow attention-derived vectors into a modeling layer (bi-LSTM) rather than collapsing to a fixed vector.
• Use a start/end span prediction mechanism for QA, trained with negative log-likelihood of true start and end indices.
• Evaluate with SQuAD (EM and F1) and CNN/DailyMail cloze tasks.

Experimental results

Research questions

• RQ1 Can bi-directional, memory-less attention between query and context improve machine comprehension over unidirectional or dynamically attended mechanisms?
• RQ2 Does maintaining token-level attention flow through subsequent modeling layers preserve information and improve answer localization?
• RQ3 How do multi-stage embeddings (character, word, contextual) contribute to QA performance?
• RQ4 What is the impact of ablations on C2Q vs Q2C attention and attention flow vs dynamic attention?

Key findings

• BiDAF achieves state-of-the-art results on SQuAD test set in both single-model and ensemble configurations (EM and F1 metrics).
• On SQuAD, BiDAF single model: EM 68.0 and F1 77.3; ensemble: EM 73.3 and F1 81.1.
• BiDAF also sets strong results on CNN/DailyMail cloze, outperforming prior single-model approaches and matching/exceeding some ensembles (CNN/DailyMail val/test).
• Ablation studies show removing C2Q or Q2C attention degrades performance; memory-less (static) attention with separate attention and modeling layers outperforms dynamic attention.
• Character and word embeddings both contribute to performance; contextual embeddings improve alignment for query words.
• Visualizations and error analyses indicate the model learns sensible query-context alignments and locations for answer spans.