[论文解读] DCN+: Mixed Objective and Deep Residual Coattention for Question Answering

Caiming Xiong, Victor W. Zhong+1 more|arXiv (Cornell University)|Oct 31, 2017

Topic Modeling参考文献 23被引用 86

一句话总结

DCN+ 将深度残差协注意力编码器与混合目标相结合，融合交叉熵损失和自我批评强化学习，在 SQuAD 上取得了最先进的结果。该模型提升了对长问题的处理能力以及整体问答性能。

ABSTRACT

Traditional models for question answering optimize using cross entropy loss, which encourages exact answers at the cost of penalizing nearby or overlapping answers that are sometimes equally accurate. We propose a mixed objective that combines cross entropy loss with self-critical policy learning. The objective uses rewards derived from word overlap to solve the misalignment between evaluation metric and optimization objective. In addition to the mixed objective, we improve dynamic coattention networks (DCN) with a deep residual coattention encoder that is inspired by recent work in deep self-attention and residual networks. Our proposals improve model performance across question types and input lengths, especially for long questions that requires the ability to capture long-term dependencies. On the Stanford Question Answering Dataset, our model achieves state-of-the-art results with 75.1% exact match accuracy and 83.1% F1, while the ensemble obtains 78.9% exact match accuracy and 86.0% F1.

研究动机与目标

• Address the gap between evaluation metrics (textual overlap) and training objective (exact span) in QA models.
• Improve representation learning for QA by deepening the coattention encoder with residual connections.
• Leverage a mixed objective that combines cross-entropy with self-critical policy learning to optimize factual overlap with ground-truth answers.
• Demonstrate performance gains across question types and lengths, especially for long questions, on SQuAD.

提出的方法

• Extend DCN with a deep residual coattention encoder to stack coattention layers and fuse them with residual connections.
• Compute a two-layer coattention and concatenate diverse representations (E1D, E2D, S1D, S2D, C1D, C2D) before final encoding.
• Introduce a mixed objective that combines cross-entropy loss over start/end positions with a reinforcement learning reward based on F1 word overlap (self-critical baseline).
• Use multi-task learning with task-dependent uncertainty weights to blend CE loss and RL loss.
• Train using ADAM and standard QA preprocessing; implement in PyTorch; use GloVe+CoVe+char n-grams embeddings.
• Evaluate on SQuAD and perform ablations to assess contributions of deep residual coattention and mixed objective.

实验结果

研究问题

• RQ1 How does a deep residual coattention encoder affect QA performance, especially for long questions?
• RQ2 Does combining cross-entropy with self-critical reinforcement learning improve alignment between optimization and evaluation metrics in QA?
• RQ3 What is the relative contribution of deep residual coattention and the mixed objective to overall QA gains?
• RQ4 Can the proposed DCN+ outperform the DCN baseline on SQuAD across different question types and lengths?

主要发现

• DCN+ achieves 75.1% EM and 83.1% F1 on SQuAD test with a single model; ensemble reaches 78.9% EM and 86.0% F1.
• DCN+ outperforms the DCN baseline with CoVe by 3.2 points in EM and 3.2 points in F1 on the development set.
• Deep residual coattention provides the largest single contribution among ablations, followed by the mixed objective.
• The mixed objective stabilizes policy learning and yields better final performance when combined with cross-entropy loss via a self-critical baseline.
• The model shows notable gains for long questions and higher-level dependencies.