[Paper Review] Technical report on Conversational Question Answering

Ying Ju, Fubang Zhao+4 more|arXiv (Cornell University)|Sep 24, 2019

Topic ModelingComputer Science20 references42 citations

TL;DR

We propose a RoBERTa-based system with rationale tagging, adversarial training, knowledge distillation and post-processing for conversational QA, achieving 90.4 F1 on CoQA test and state-of-the-art single-model performance.

ABSTRACT

Conversational Question Answering is a challenging task since it requires understanding of conversational history. In this project, we propose a new system RoBERTa + AT +KD, which involves rationale tagging multi-task, adversarial training, knowledge distillation and a linguistic post-process strategy. Our single model achieves 90.4(F1) on the CoQA test set without data augmentation, outperforming the current state-of-the-art single model by 2.6% F1.

Motivation & Objective

• Motivate improved fine-tuning for conversational QA by leveraging rationale information from answers.
• Introduce multi-task rationale tagging to guide extraction.
• Incorporate adversarial and virtual adversarial training to boost robustness.
• Apply knowledge distillation to transfer richer signals from multiple teacher models.
• Evaluate on CoQA to establish state-of-the-art results and analyze extractive model headroom.

Proposed method

• Baseline RoBERTa setup for CoQA with concatenated history-aware input.
• Rationale tagging multi-task to predict tokens included in the rationale and integrate this into Yes/No/Unk decisions.
• Adversarial training (AT) by perturbing word embeddings and Virtual Adversarial Training (VAT) for unsupervised perturbations.
• Knowledge Distillation (KD) using multiple teacher models to provide soft targets to a student model.
• Post-processing with a word-similarity based approach to handle non-extractive/alternative-form answers and ensemble via GA-guided model selection.
• Ensemble of multiple models using averaged logits, optimized by a genetic algorithm under size constraints.

Experimental results

Research questions

• RQ1 Can rationale tagging and adversarial/knowledge-distilled training improve performance on conversational QA without data augmentation?
• RQ2 What is the practical upper bound for extractive models on CoQA given free-form answers?
• RQ3 How do post-processing and ensembling affect CoQA performance compared to a single model?

Key findings

• Single model achieves 90.4 F1 on CoQA test (state-of-the-art among single models).
• Ablation study shows: baseline 89.5 F1; +Rationale Tagging 90.0; +Adversarial Training 90.7; +Knowledge Distillation 91.0; +Post-Processing 91.3.
• Best single-model setup reaches 91.3 F1 on dev with post-processing; ensemble further improves to 90.7 F1 on test (and 91.8 on dev) under a constrained ensemble.
• Ensembling 9 models via GA achieves 91.5 F1 on CoQA test, outperforming simple averaging (91.2).
• Upper-bound analysis suggests 93.0 F1 with first human reference and 95.1 F1 with all four human references, indicating extractive headroom limits (4.9%).