[Paper Review] EfficientQA : a RoBERTa Based Phrase-Indexed Question-Answering System

Sofian Chaybouti, Achraf Saghe+1 more|arXiv (Cornell University)|Jan 6, 2021

Topic Modeling19 references4 citations

TL;DR

EfficientQA proposes a RoBERTa-based phrase-indexed question answering system that uses question-agnostic candidate extraction and siamese network training to map questions and answer candidates into a shared dense vector space, enabling efficient nearest-neighbor search. It achieves state-of-the-art results on the PIQA benchmark, outperforming prior methods by 1.3 points in exact-match and 1.4 points in F1-score while using a smaller RoBERTa-base model instead of BERT-large.

ABSTRACT

State-of-the-art extractive question-answering models achieve superhuman performances on the SQuAD benchmark. Yet, they are unreasonably heavy and need expensive GPU computing to answer questions in a reasonable time. Thus, they cannot be used in the open-domain question-answering paradigm for real-world queries on hundreds of thousands of documents. In this paper, we explore the possibility of transferring the natural language understanding of language models into dense vectors representing questions and answer candidates to make question-answering compatible with a simple nearest neighbor search task. This new model, which we call EfficientQA, takes advantage of the pair of sequences kind of input of BERT-based models to build meaningful, dense representations of candidate answers. These latter are extracted from the context in a question-agnostic fashion. Our model achieves state-of-the-art results in Phrase-Indexed Question Answering (PIQA), beating the previous state-of-art by 1.3 points in exact-match and 1.4 points in f1-score. These results show that dense vectors can embed rich semantic representations of sequences, although these were built from language models not originally trained for the use case. Thus, to build more resource-efficient NLP systems in the future, training language models better adapted to build dense representations of phrases is one of the possibilities.

Motivation & Objective

• To address the scalability limitations of heavy extractive QA models in open-domain settings by enabling efficient inference via dense vector indexing.
• To improve efficiency in open-domain question answering by decoupling candidate extraction from question encoding, reducing the search space.
• To demonstrate that dense representations from pre-trained language models can effectively encode phrase-level semantics for nearest-neighbor retrieval.
• To close the performance gap between efficient phrase-indexed models and full-context, question-aware QA systems.
• To enable deployment of QA systems on CPU-only or low-resource environments through lightweight indexing and inference.

Proposed method

• Extract candidate answers from documents in a question-agnostic manner using a RoBERTa-based model trained to predict start and end positions jointly.
• Use a siamese network architecture to map both questions and candidate answers into a shared dense vector space.
• Train the model using contrastive learning with triplet loss to maximize semantic similarity between question-answer pairs and minimize similarity with negative pairs.
• Index all candidate answer vectors offline, enabling online inference via simple nearest-neighbor search using cosine similarity.
• Leverage RoBERTa-base’s contextual representations to generate high-quality dense vectors without requiring question-aware fine-tuning during indexing.
• Apply mixed-precision training and gradient accumulation to reduce training time and memory usage on a single 24GB GPU.

Experimental results

Research questions

• RQ1 Can dense vector representations from pre-trained language models effectively encode phrase-level semantics for efficient nearest-neighbor retrieval in question answering?
• RQ2 Does question-agnostic candidate extraction reduce the search space sufficiently to improve retrieval accuracy despite lack of question context during indexing?
• RQ3 Can a smaller RoBERTa-base model outperform larger BERT-large models in phrase-indexed QA when using optimized dense representation learning?
• RQ4 To what extent does the joint modeling of start and end position probabilities improve candidate quality compared to independent prediction?
• RQ5 How does the performance gap between efficient phrase-indexed models and full-context QA models vary across languages with different training data sizes?

Key findings

• EfficientQA achieves 74.9% exact-match and 83.1% F1-score on the PIQA benchmark, outperforming the previous SOTA DENSPI by 1.3 and 1.4 points respectively.
• The proposed architecture for candidate extraction, which models start and end positions jointly, yields 92.3% exact-match and 96.7% F1-score for 100 candidates, significantly outperforming the independent prediction baseline (54.1% EM, 72.4% F1 with beam search).
• Despite using a smaller RoBERTa-base model (125M parameters), EfficientQA surpasses DENSPI, which uses a larger BERT-large model (340M parameters), demonstrating the efficiency of the method.
• On the FQuAD benchmark, EfficientQA achieves 64.4% exact-match and 76.0% F1-score in French, showing a performance gap of 13.2 points in EM and 11.3 points in F1 compared to a fine-tuned CamemBERT model, indicating room for improvement in low-resource languages.
• The model trains in approximately one week on a single 24GB GPU using mixed-precision and gradient accumulation, with a batch size equivalent to 32, showing low training resource demands.
• The system enables efficient inference via nearest-neighbor search, reducing the need for expensive GPU inference during online deployment.