[论文解读] Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?

Neeraj Varshney, Man Luo+1 more|arXiv (Cornell University)|Nov 23, 2022

Topic Modeling被引用 10

一句话总结

本文提出一种动态的、类人知识阅读策略，用于开放域问答任务，结合闭卷推理与迭代式、基于置信度的外部知识检索。通过仅自适应地阅读必要段落——从少量开始，仅在需要时逐步增加——该方法在NQ Open数据集上达到最先进准确率（最高55.10%），同时相比最先进模型Fusion-in-Decoder，推理成本降低81.68%。

ABSTRACT

Recent state-of-the-art open-domain QA models are typically based on a two stage retriever-reader approach in which the retriever first finds the relevant knowledge/passages and the reader then leverages that to predict the answer. Prior work has shown that the performance of the reader usually tends to improve with the increase in the number of these passages. Thus, state-of-the-art models use a large number of passages (e.g. 100) for inference. While the reader in this approach achieves high prediction performance, its inference is computationally very expensive. We humans, on the other hand, use a more efficient strategy while answering: firstly, if we can confidently answer the question using our already acquired knowledge then we do not even use the external knowledge, and in the case when we do require external knowledge, we don't read the entire knowledge at once, instead, we only read that much knowledge that is sufficient to find the answer. Motivated by this procedure, we ask a research question "Can the open-domain QA reader utilize external knowledge efficiently like humans without sacrificing the prediction performance?" Driven by this question, we explore an approach that utilizes both 'closed-book' (leveraging knowledge already present in the model parameters) and 'open-book' inference (leveraging external knowledge). Furthermore, instead of using a large fixed number of passages for open-book inference, we dynamically read the external knowledge in multiple 'knowledge iterations'. Through comprehensive experiments on NQ and TriviaQA datasets, we demonstrate that this dynamic reading approach improves both the 'inference efficiency' and the 'prediction accuracy' of the reader. Comparing with the FiD reader, this approach matches its accuracy by utilizing just 18.32% of its reader inference cost and also outperforms it by achieving up to 55.10% accuracy on NQ Open.

研究动机与目标

• 解决依赖固定大量检索段落的最先进开放域问答阅读器所面临的高推理成本问题。
• 探究阅读器是否可通过动态仅读取必要知识，在显著降低计算成本的同时实现高预测准确率。
• 探索结合迭代式知识获取的混合闭卷/开卷推理策略，能否同时提升效率与准确率。
• 建模类人推理过程：首先基于内部知识尝试作答，仅在置信度较低时逐步访问外部知识。

提出的方法

• 使用闭卷阅读器模型（如T5-large）首先仅基于模型参数中嵌入的知识进行答案预测。
• 若预测置信度低于阈值，则启动开卷推理，使用外部知识检索器。
• 应用迭代式知识阅读：从少量段落开始（如S₁=10），仅在置信度仍低时逐步增加数量（S₂=20，S₃=100）。
• 使用置信度分数作为停止准则，避免过度依赖过多段落。
• 通过级联系统组合多次知识迭代的预测结果，基于置信度选择最佳输出。
• 以FLOPs衡量效率，并在NQ和TriviaQA数据集上使用精确匹配（EM）与AUC指标评估性能。

实验结果

研究问题

• RQ1 阅读器模型是否能在使用远少于标准方法的外部段落数量下，实现高预测准确率？
• RQ2 与固定大规模检索相比，通过多轮迭代动态读取知识是否能同时提升效率与准确率？
• RQ3 混合闭卷/开卷推理策略是否能在不牺牲答案质量的前提下降低计算成本？
• RQ4 相较于单次大规模检索，使用多轮知识迭代（如K=2或K=3）是否具有可测量的优势，即使总段落数相同？
• RQ5 该方法在使用远少于FLOPs的情况下，能否在性能上匹配或超越最先进模型Fusion-in-Decoder（FiD）？

主要发现

• 所提方法在NQ Open基准上达到最高55.10%的精确匹配准确率，优于所有对比基线模型，包括FiD、RAG与DPR。
• 在与FiD（base）模型相同的精确匹配准确率（20.19×10¹¹ FLOPs）下，所提系统将推理成本降低至仅3.69×10¹¹ FLOPs，降幅达81.68%。
• 对于FiD（large）模型（70.69×10¹¹ FLOPs），所提方法在20.59×10¹¹ FLOPs下实现相当的准确率，成本降低71.1%。
• 采用两轮知识迭代（K=2）的级联系统，AUC达到46.25，高于总段落数相同的单步检索（43.56），证明了迭代阅读的优势。
• 在TriviaQA上，该方法达到72.33%的准确率，同样优于现有方法，且保持高效率。
• 在相同FLOP成本下，该系统相比基线开卷模型准确率更高：例如在0.2×10¹¹ FLOPs下，准确率达48.98%，高于OB₁的46.78%。