[Paper Review] QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension

Adams Wei Yu, D. Dohan+5 more|arXiv (Cornell University)|Apr 23, 2018

Topic Modeling440 citations

TL;DR

QANet removes recurrent networks by using a feedforward architecture with convolution and self-attention, achieving state-of-the-art accuracy on SQuAD while being several times faster to train and run inference, with data augmentation via backtranslation further boosting results.

ABSTRACT

Current end-to-end machine reading and question answering (Q\\&A) models are primarily based on recurrent neural networks (RNNs) with attention. Despite their success, these models are often slow for both training and inference due to the sequential nature of RNNs. We propose a new Q\\&A architecture called QANet, which does not require recurrent networks: Its encoder consists exclusively of convolution and self-attention, where convolution models local interactions and self-attention models global interactions. On the SQuAD dataset, our model is 3x to 13x faster in training and 4x to 9x faster in inference, while achieving equivalent accuracy to recurrent models. The speed-up gain allows us to train the model with much more data. We hence combine our model with data generated by backtranslation from a neural machine translation model. On the SQuAD dataset, our single model, trained with augmented data, achieves 84.6 F1 score on the test set, which is significantly better than the best published F1 score of 81.8.

Motivation & Objective

• Motivate fast reading comprehension models that avoid RNNs due to slow training/inference times.
• Propose a encoder architecture built from convolutions and self-attention to capture local and global text interactions.
• Show that speedups enable larger-scale data augmentation to improve accuracy on SQuAD.

Proposed method

• Encoder blocks composed of depthwise separable convolutions, multi-head self-attention, and feed-forward layers with residual connections and layer normalization.
• Context-query attention using a trilinear similarity and additional query-to-context attention.
• Shared encoder weights for context and question, with seven model encoder blocks and two within-encoder convolution blocks.
• Output layer predicting start and end positions via softmax distributions over context positions.
• Data augmentation via backtranslation (English↔French/English↔German) to create paraphrastic training examples and improve generalization.

Experimental results

Research questions

• RQ1 Can a non-recurrent encoder based solely on convolution and self-attention achieve competitive or superior accuracy on SQuAD compared to RNN-based models?
• RQ2 How much speedup in training and inference can be obtained without sacrificing accuracy?
• RQ3 Does backtranslation-based data augmentation improve QA performance on SQuAD, and what data mix yields best results?

Key findings

• QANet achieves comparable accuracy to recurrent models on SQuAD while offering significant speedups (3x–13x training, 4x–9x inference).
• On SQuAD, a single QANet model trained with augmented data reaches 84.6 F1 on the test set, exceeding prior bests.
• The model with backtranslated augmentation (×3) yields the best dev/test results reported (e.g., test EM/F1 76.2/84.6).
• Ablation shows removing convolutions or self-attention degrades performance, confirming their complementary roles in capturing local vs global interactions.
• Data augmentation improves performance, with optimal sampling between original and augmented data yielding noticeable gains.