[论文解读] ERAGent: Enhancing Retrieval-Augmented Language Models with Improved Accuracy, Efficiency, and Personalization

Yunxiao Shi, Xing Zi+4 more|arXiv (Cornell University)|May 6, 2024

Topic Modeling被引用 5

一句话总结

ERAGent 引入 Enhanced Question Rewriter、Knowledge Filter、Retrieval Trigger，以及通过 Experiential Learner 的个性化，在基于 RAG 的问答中提升检索质量、效率与对用户的定制化回答，并在多个数据集上进行了广泛评估。

ABSTRACT

Retrieval-augmented generation (RAG) for language models significantly improves language understanding systems. The basic retrieval-then-read pipeline of response generation has evolved into a more extended process due to the integration of various components, sometimes even forming loop structures. Despite its advancements in improving response accuracy, challenges like poor retrieval quality for complex questions that require the search of multifaceted semantic information, inefficiencies in knowledge re-retrieval during long-term serving, and lack of personalized responses persist. Motivated by transcending these limitations, we introduce ERAGent, a cutting-edge framework that embodies an advancement in the RAG area. Our contribution is the introduction of the synergistically operated module: Enhanced Question Rewriter and Knowledge Filter, for better retrieval quality. Retrieval Trigger is incorporated to curtail extraneous external knowledge retrieval without sacrificing response quality. ERAGent also personalizes responses by incorporating a learned user profile. The efficiency and personalization characteristics of ERAGent are supported by the Experiential Learner module which makes the AI assistant being capable of expanding its knowledge and modeling user profile incrementally. Rigorous evaluations across six datasets and three question-answering tasks prove ERAGent's superior accuracy, efficiency, and personalization, emphasizing its potential to advance the RAG field and its applicability in practical systems.

研究动机与目标

• 解决标准 RAG 的局限性，包括对复杂查询的检索质量、长期服务中的冗余再检索，以及缺乏个性化的问题。
• 提出机制：Enhanced Question Rewriter、Retrieval Trigger、Knowledge Filter、Personalized LLM Reader，以及 Experiential Learner。
• 通过对多种问答任务和数据集的全面实验，证明在准确性、效率和个性化方面的提升。

提出的方法

• 引入 Enhanced Question Rewriter 以产生更清晰、粒度更细的查询，从而获得更好的检索。
• 实现 Retrieval Trigger，根据历史上下文和知识边界来决定何时应检索外部知识。
• 应用 Knowledge Filter，利用 NLI 仅保留蕴涵式相关检索知识。
• 使用 Personalized LLM Reader，在提示中纳入用户画像以实现定制化回答。
• 利用 Experiential Learner 通过交互构建 Memory Knowledge Database 和动态用户画像，以提升效率和个性化。
• 在单轮开放域问答、单轮多跳问答，以及跨六个数据集的多轮多会话问答上评估 ERAGent。

实验结果

研究问题

• RQ1 增强型问题改写器是否在提高答案准确性方面优于传统问题改写器？
• RQ2 知识过滤器是否能有效过滤不相关上下文以提升回答质量？
• RQ3 在多轮会话设置中，个性化回答是否优于非个性化回答？
• RQ4 Experiential Learner 是否在不降低质量的前提下提升检索效率？

主要发现

Method	Dataset	EM	Precision	Recall	Hit Rate
Standard	NQ	38.00	54.21	77.38	55.00
Rewriter	NQ	35.50	55.16	73.48	54.00
Rewriter+	NQ	38.00	57.24	73.93	55.50
Filter	NQ	36.00	53.82	76.89	52.50
Rewriter+Filter	NQ	40.50	56.84	75.50	58.50
Standard	PopQA	29.50	68.43	44.38	35.00
Rewriter	PopQA	30.00	64.46	46.50	35.00
Rewriter+	PopQA	32.00	69.54	47.59	37.50
Filter	PopQA	34.00	70.17	46.17	38.00
Rewriter+Filter	PopQA	36.00	69.13	48.40	40.50
Standard	AmbigNQ	38.00	62.03	64.45	52.00
Rewriter	AmbigNQ	41.00	65.35	63.78	55.50
Rewriter+	AmbigNQ	45.50	67.70	65.84	58.50
Filter	AmbigNQ	39.50	65.18	64.88	55.00
Rewriter+Filter	AmbigNQ	47.00	69.82	67.06	63.50

• 增强型 Question Rewriter 与 Knowledge Filter 的组合（Rewriter+Filter）在单轮问答数据集上通常实现最佳整体性能。
• 增强型 Question Rewriter 相较于基线持续提升性能，尤其在更难/更苛刻的数据集上。
• Knowledge Filter 在 PopQA 和 AmbigNQ 上提升了性能，但在 NQ 上因严格的蕴涵性标准而有所下降。
• 在多跳推理任务中，Rewriter+Filter 提供最强的增益，凸显改写与过滤的协同效果。
• 多轮会话实验表明，通过用户画像的个性化与记忆增强可以提升回答质量与效率，相较于非个性化基线。