[Paper Review] BAS: An Answer Selection Method Using BERT Language Model

Jamshid Mozafari, Afsaneh Fatemi+1 more|arXiv (Cornell University)|Nov 4, 2019

Topic Modeling30 references15 citations

TL;DR

This paper proposes BAS (BERT Answer Selection), a novel answer selection method that leverages the BERT language model to improve semantic and syntactic understanding of questions and candidate answers. Experiments on TrecQA Raw, TrecQA Clean, and WikiQA datasets show that BAS significantly outperforms prior methods, demonstrating that robust language modeling enhances answer selection performance, especially when combined with an effective classifier.

ABSTRACT

In recent years, Question Answering systems have become more popular and widely used by users. Despite the increasing popularity of these systems, the their performance is not even sufficient for textual data and requires further research. These systems consist of several parts that one of them is the Answer Selection component. This component detects the most relevant answer from a list of candidate answers. The methods presented in previous researches have attempted to provide an independent model to undertake the answer-selection task. An independent model cannot comprehend the syntactic and semantic features of questions and answers with a small training dataset. To fill this gap, language models can be employed in implementing the answer selection part. This action enables the model to have a better understanding of the language in order to understand questions and answers better than previous works. In this research, we will present the "BAS" (BERT Answer Selection) that uses the BERT language model to comprehend language. The empirical results of applying the model on the TrecQA Raw, TrecQA Clean, and WikiQA datasets demonstrate that using a robust language model such as BERT can enhance the performance. Using a more robust classifier also enhances the effect of the language model on the answer selection component. The results demonstrate that language comprehension is an essential requirement in natural language processing tasks such as answer-selection.

Motivation & Objective

• To address the limitations of existing answer selection models that struggle with semantic and syntactic understanding, especially on small datasets.
• To explore whether pre-trained language models like BERT can enhance answer selection by improving contextual understanding of questions and candidate answers.
• To develop a unified framework that integrates BERT with a robust classifier to maximize performance on answer selection tasks.
• To evaluate the effectiveness of BERT-based modeling in low-resource settings common in answer selection.
• To demonstrate that language comprehension is a critical component for high-performing answer selection in NLP.

Proposed method

• The BAS framework fine-tunes the BERT base model on answer selection tasks, leveraging its contextual representations for both questions and candidate answers.
• Input sequences are constructed by concatenating the question and each candidate answer with [CLS] and [SEP] tokens for BERT input.
• The [CLS] token’s final hidden state is used as the sentence-level representation for classification.
• A feed-forward classifier head is added on top of the BERT encoder to predict the relevance score of each candidate answer.
• The model is trained end-to-end using cross-entropy loss to optimize for answer relevance.
• The approach is evaluated on three benchmark datasets: TrecQA Raw, TrecQA Clean, and WikiQA.

Experimental results

Research questions

• RQ1 Can BERT-based models significantly improve answer selection performance compared to traditional methods?
• RQ2 How does the integration of a robust language model like BERT affect the model's ability to capture syntactic and semantic features in questions and answers?
• RQ3 Does the performance gain from BERT depend on the quality of the classifier used on top of the embeddings?
• RQ4 How does BAS perform on datasets with limited training data?
• RQ5 To what extent does language comprehension enhance answer selection in open-domain QA systems?

Key findings

• BAS achieves state-of-the-art performance on the TrecQA Raw, TrecQA Clean, and WikiQA datasets, outperforming previous methods.
• The use of BERT significantly improves the model's ability to understand semantic and syntactic relationships between questions and answers.
• Combining BERT with a strong classifier leads to a more substantial performance gain than using BERT alone.
• The model demonstrates robustness even on smaller training sets, indicating effective transfer learning.
• Empirical results confirm that language comprehension is a critical factor in achieving high accuracy in answer selection tasks.
• The model's performance is consistently better than prior independent models, especially in complex semantic matching scenarios.