[Paper Review] A BERT Baseline for the Natural Questions

Chris Alberti, Kenton Lee+1 more|arXiv (Cornell University)|Jan 24, 2019

Topic Modeling9 references97 citations

TL;DR

A BERT-based single-model baseline for Natural Questions that jointly predicts short and long answers, using sliding windows and downsampling of null instances to improve F1 scores relative to previous baselines.

ABSTRACT

This technical note describes a new baseline for the Natural Questions. Our model is based on BERT and reduces the gap between the model F1 scores reported in the original dataset paper and the human upper bound by 30% and 50% relative for the long and short answer tasks respectively. This baseline has been submitted to the official NQ leaderboard at ai.google.com/research/NaturalQuestions. Code, preprocessed data and pretrained model are available at https://github.com/google-research/language/tree/master/language/question_answering/bert_joint.

Motivation & Objective

• Motivate Natural Questions as a harder QA benchmark and establish a strong BERT-based baseline.
• Develop a single model that jointly predicts short and long answers in NQ.
• Improve training efficiency and effectiveness via data preprocessing and sampling strategies.
• Demonstrate substantial improvement over previous NQ baselines and approach human performance margins.

Proposed method

• Initialize from a BERT model finetuned on SQuAD 1.1.
• Create training instances by sliding a 512-token window over documents with stride 128.
• Downsample null (no-answer) instances by 50x to balance training data.
• Introduce atomic markup tokens [Paragraph=N], [Table=N], [List=N] to signal document structure to the model.
• Predict start, end, and answer-type (short/long/yes/no/no-answer) jointly in a single model.
• Rank spans by a score g(c,s,e) = f_start(s,c) + f_end(e,c) - f_start([CLS],c) - f_end([CLS],c).
• Limit predictions to a single short answer and rely on evaluation script to handle long/no-answer adjustments.

Experimental results

Research questions

• RQ1 Can a single BERT model jointly predict short and long answers effectively for Natural Questions?
• RQ2 Do windowing, null-downsampling, and structural markup improve QA performance on NQ compared to previous baselines?
• RQ3 What is the impact of training with a joint start/end/type objective on NQ tasks (short/long/yes/no/no-answer)?

Key findings

• The BERT joint model substantially outperforms prior NQ baselines and narrows the gap to the human upper bound by 30% for long answers and 50% for short answers.
• Training uses a balanced mix of non-null and downsampled null instances, enabling effective learning despite many nulls.
• The model achieves strong dev/test F1 gains over baselines such as DocumentQA, DecAtt + DocReader, and prior NQ baselines.
• The approach still leaves a notable headroom (over 20 F1 points) for both long and short answer tasks, indicating room for further improvements.