[Paper Review] Attention-Guided Answer Distillation for Machine Reading Comprehension

Minghao Hu, Yuxing Peng+5 more|arXiv (Cornell University)|Aug 23, 2018

Topic Modeling29 references14 citations

TL;DR

This paper proposes attention-guided answer distillation to compress ensemble machine reading comprehension (MRC) models into a single, efficient student model while improving robustness. By applying vanilla knowledge distillation, answer distillation (to penalize confusing answers), and attention distillation (to align attention distributions), the student model achieves 12× faster inference with only a 0.4% F1 drop on SQuAD and outperforms the teacher on adversarial SQuAD and NarrativeQA benchmarks.

ABSTRACT

Despite that current reading comprehension systems have achieved significant advancements, their promising performances are often obtained at the cost of making an ensemble of numerous models. Besides, existing approaches are also vulnerable to adversarial attacks. This paper tackles these problems by leveraging knowledge distillation, which aims to transfer knowledge from an ensemble model to a single model. We first demonstrate that vanilla knowledge distillation applied to answer span prediction is effective for reading comprehension systems. We then propose two novel approaches that not only penalize the prediction on confusing answers but also guide the training with alignment information distilled from the ensemble. Experiments show that our best student model has only a slight drop of 0.4% F1 on the SQuAD test set compared to the ensemble teacher, while running 12x faster during inference. It even outperforms the teacher on adversarial SQuAD datasets and NarrativeQA benchmark.

Motivation & Objective

• To address the inefficiency and vulnerability to adversarial attacks in state-of-the-art MRC systems, which often rely on ensembles of multiple models.
• To investigate knowledge distillation as a method to compress ensemble models into a single, efficient, and robust student model.
• To mitigate biased distillation, where teacher models' overconfident predictions on confusing answers mislead the student.
• To improve student generalization and robustness by transferring not only output distributions but also attention-based alignment and distractor-aware supervision.

Proposed method

• Apply vanilla knowledge distillation by minimizing cross-entropy between soft label distributions of answer spans from an ensemble teacher and a single student model.
• Introduce answer distillation that penalizes the student’s prediction on the most confusing answer span by using a margin loss, based on the teacher’s strongest distractor prediction.
• Propose attention distillation to match the attention distributions between teacher and student models using mean-squared error, improving alignment and interpretability.
• Use the ensemble model’s attention maps and answer boundary probabilities to guide the student’s training, enhancing both efficiency and robustness.
• Combine all three distillation components—output distillation, answer distillation, and attention distillation—into a unified training objective.
• Train the student model end-to-end using a weighted combination of cross-entropy loss for answer spans, margin loss for confusing answers, and MSE loss for attention alignment.

Experimental results

Research questions

• RQ1 Can knowledge distillation effectively compress ensemble MRC models into a single model while preserving performance?
• RQ2 How can biased distillation—where teachers overconfidently predict semantically contradictory distractors—be mitigated during distillation?
• RQ3 Can attention distillation improve student generalization by transferring alignment patterns from the teacher?
• RQ4 Does distillation improve robustness against adversarial examples, particularly in out-of-distribution settings?
• RQ5 Can a single distilled model outperform the original ensemble teacher in terms of both efficiency and robustness?

Key findings

• The best student model achieves only a 0.4% F1 drop on the SQuAD test set compared to the ensemble teacher, while running 12× faster during inference.
• The distilled student model outperforms the teacher model on the adversarial SQuAD dataset, demonstrating improved robustness against adversarial examples.
• On the NarrativeQA benchmark, the student model surpasses the teacher in Bleu-1 score, indicating better generation quality on open-domain questions.
• Answer distillation effectively reduces the student’s confidence in confusing answer spans, preventing overconfident wrong predictions caused by biased teacher signals.
• Attention distillation improves the student’s ability to align question and passage tokens, leading to more precise and interpretable attention patterns.
• The combination of all three distillation components yields the best overall performance, showing that complementary supervision from outputs, confusing answers, and attention enhances model generalization.