[Paper Review] Adversarial Examples for Evaluating Reading Comprehension Systems

Robin Jia, Percy Liang|arXiv (Cornell University)|Jul 23, 2017

Topic Modeling30 references262 citations

TL;DR

The paper introduces adversarial evaluation for SQuAD by appending distractor sentences to paragraphs, showing that16 models drop from ~75 F1 to ~36 F1 (and to ~7 F1 with ungrammatical distractors), revealing limited true language understanding.

ABSTRACT

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of $75\%$ F1 score to $36\%$; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to $7\%$. We hope our insights will motivate the development of new models that understand language more precisely.

Motivation & Objective

• Assess whether SQuAD models rely on superficial patterns rather than true language understanding.
• Develop adversarial methods that preserve the correct answer while confusing models.
• Quantify robustness of multiple models to adversarially inserted sentences.
• Provide public code and data to enable standardized adversarial evaluation.

Proposed method

• Define adversarial framework A that transforms (p, q, a) into (p', q', a') with compatible but misleading additions.
• Create concatenative adversaries AddSent and AddOneSent that append sentences to the paragraph.
• Design AddAny (and AddCommon) to insert arbitrary or common-word sentences via search strategies.
• Automatically generate distractor sentences through semantic perturbations and type-matched fake answers, then crowdsource grammatical fixes.
• Evaluate models on standard vs adversarial inputs across multiple architectures (BiDAF, Match-LSTM, and others).
• Provide reproducible code and data for running AddSent/AddAny on SQuAD systems.

Experimental results

Research questions

• RQ1 Do current reading comprehension models exhibit true language understanding or rely on superficial cues?
• RQ2 How robust are models to adversarially inserted sentences that do not contradict the correct answer?
• RQ3 Are adversarial vulnerabilities transferable across different models and architectures?
• RQ4 Can training on adversarial examples improve robustness, and under what conditions?

Key findings

• Adversarial perturbations drastically reduce model performance: average F1 drops from ~75% to ~36% with AddSent, and to ~7% with ungrammatical AddAny-type inputs.
• All sixteen evaluated models are vulnerable; some transferability of adversarial examples occurs across models, especially for AddSent.
• Humans show substantially higher robustness than models on AddSent (AddOneSent shows smaller human drops).
• Training on adversarial data provides limited robustness improvements and often transfers poorly to altered adversaries (AddSentMod).
• Adversarial examples tend to cause models to predict spans within adversarial sentences in many failures, indicating over-stability to semantics-altering edits.