[Paper Review] QuAC : Question Answering in Context

Eunsol Choi, He He+6 more|arXiv (Cornell University)|Aug 21, 2018

Topic ModelingComputer Science22 references53 citations

TL;DR

QUAC introduces a large-scale dataset of information-seeking dialog QA where a student asks questions about a hidden Wikipedia section and a teacher answers with text spans, revealing open-ended and unanswerable cases and requiring dialog context for accurate answering.

ABSTRACT

We present QuAC, a dataset for Question Answering in Context that contains 14K information-seeking QA dialogs (100K questions in total). The dialogs involve two crowd workers: (1) a student who poses a sequence of freeform questions to learn as much as possible about a hidden Wikipedia text, and (2) a teacher who answers the questions by providing short excerpts from the text. QuAC introduces challenges not found in existing machine comprehension datasets: its questions are often more open-ended, unanswerable, or only meaningful within the dialog context, as we show in a detailed qualitative evaluation. We also report results for a number of reference models, including a recently state-of-the-art reading comprehension architecture extended to model dialog context. Our best model underperforms humans by 20 F1, suggesting that there is significant room for future work on this data. Dataset, baseline, and leaderboard available at http://quac.ai.

Motivation & Objective

• Address information-seeking dialog where questions are context-dependent and may be unanswerable from the section alone.
• Create a large-scale dataset (14K dialogs, 100K questions) with teacher-student interactions and access asymmetry.
• Investigate how dialog context affects QA performance and provide strong baselines that incorporate context.

Proposed method

• Crowd-sourced teacher-student dialogs around a Wikipedia section (student sees only title and first paragraph; teacher sees full section) where answers are spans within the section text (max 30 tokens).
• Teacher outputs an answer span plus dialog acts (continuation, affirmation, answerability) to guide the dialog.
• Dataset collection uses Mechanical Turk with quality controls and rewards per turn; dialogs terminate after 12 answered questions or early stop conditions.
• Evaluation treats span prediction as in SQuAD-style F1 with unanswerable handling, plus a human-equivalence score (HEQ) comparing system to human performance.
• Baseline models include text-similarity, a BiDAF++ style reading comprehension model adapted for dialog context, and a context-augmented variant that conditions on previous turns.

Experimental results

Research questions

• RQ1 How does dialog context influence answerability and span localization in information-seeking QA?
• RQ2 What portion of questions are contextual or non-factoid, and how do unanswerable questions affect modeling?
• RQ3 Can state-of-the-art reading comprehension models extended to dialog context approach human performance on QUAC?
• RQ4 What is the upper-bound performance achievable by humans and how far do current models fall from it?

Key findings

• QUAC contains 14K dialogs (100K questions) with many open-ended, contextual, and unanswerable questions.
• A strong context-aware baseline (BiDAF++ with dialog context) improves over context-agnostic models but still lags behind human performance by about 20 F1 points on the full task.
• Context incorporation (up to 2-3 previous turns) yields significant gains; further context shows diminishing returns.
• Many questions are contextual (86%), with a substantial portion relying on dialog history (44% refer to dialog history; 61% refer to article subject).
• Average answer length is longer than in standard SQuAD-style tasks (about 15 tokens), reflecting open-ended responses.
• Human upper bound on F1 is 80.8, while the best model reaches around 60 F1 and approximately 60% HEQ-D, illustrating substantial room for improvement.