[Paper Review] Open-Domain Conversational Question Answering with Historical Answers

Hung-Chieh Fang, Kuo-Han Hung+2 more|arXiv (Cornell University)|Nov 17, 2022

Topic Modeling4 citations

TL;DR

This paper proposes ConvADR-QA, a novel open-domain conversational question answering framework that enhances dense retrieval and answer generation by incorporating historical answers as direct input signals. By leveraging predicted answers from a reader model to improve passage retrieval, the method achieves new state-of-the-art performance on the OR-QuAC benchmark in both extractive and generative settings, demonstrating that historical answers significantly boost both retrieval and answering accuracy.

ABSTRACT

Open-domain conversational question answering can be viewed as two tasks: passage retrieval and conversational question answering, where the former relies on selecting candidate passages from a large corpus and the latter requires better understanding of a question with contexts to predict the answers. This paper proposes ConvADR-QA that leverages historical answers to boost retrieval performance and further achieves better answering performance. In our proposed framework, the retrievers use a teacher-student framework to reduce noises from previous turns. Our experiments on the benchmark dataset, OR-QuAC, demonstrate that our model outperforms existing baselines in both extractive and generative reader settings, well justifying the effectiveness of historical answers for open-domain conversational question answering.

Motivation & Objective

• To improve open-domain conversational question answering by incorporating historical answers as explicit signals in the retrieval process.
• To address the limitations of relying solely on historical questions in multi-turn dialogue systems, which may fail to capture semantic continuity.
• To enhance retrieval performance in open-domain settings where millions of candidate passages exist and dense retrieval is essential.
• To demonstrate that predicted answers—rather than just questions—can serve as effective supervision signals for better passage retrieval.
• To achieve state-of-the-art performance in both extractive and generative answer generation settings using a simple yet effective design.

Proposed method

• The model extends dense retrieval by incorporating predicted historical answers into the query representation during passage retrieval.
• It uses a two-stage pipeline: first, a dense retriever retrieves relevant passages using a query enriched with historical answers; second, a reader model (either extractive or generative) predicts answers from retrieved passages.
• The reader model (e.g., BERT-based extractive reader or FiD for generative QA) generates answers, which are then used as input signals for the next turn’s retrieval.
• The method applies knowledge distillation principles by using the reader’s output to refine the retriever’s input, effectively propagating answer signals forward.
• The approach avoids complex graph structures or additional parameters, relying instead on direct integration of predicted answers into the retrieval query.
• It is compatible with existing strong baselines such as ORConvQA and FiD, enabling plug-and-play improvements.

Experimental results

Research questions

• RQ1 Can historical answers improve passage retrieval performance in open-domain conversational question answering?
• RQ2 Does incorporating predicted answers as input signals lead to better answer generation compared to relying only on historical questions?
• RQ3 How does the quality of predicted answers affect the overall retrieval and answering performance?
• RQ4 Can a simple integration of answer signals outperform more complex methods like graph-guided retrieval?
• RQ5 What is the impact of error propagation from low-quality answer predictions on retrieval effectiveness?

Key findings

• ConvADR-QA achieves new state-of-the-art performance on the OR-QuAC benchmark in both extractive and generative QA settings.
• The model outperforms all baselines in retrieval MRR@5 and answer generation metrics, demonstrating the effectiveness of using historical answers as retrieval signals.
• Using predicted answers improves retrieval performance over baselines that use only historical questions, with MRR@5 increasing significantly.
• The oracle setting using gold historical answers achieves higher performance than predicted answers, confirming that answer quality directly impacts retrieval success.
• Error propagation is observed: when a weak reader model (e.g., FiD) is used, retrieval performance drops, indicating that answer quality affects retrieval accuracy.
• Qualitative analysis shows that historical answers directly influence answer predictions, with ConvADR-QA correctly generating answers that align with prior responses, unlike ConvDR which often fails to maintain consistency.