[论文解读] Rethinking Evaluation in Retrieval-Augmented Personalized Dialogue: A Cognitive and Linguistic Perspective

Tianyi Zhang, David Traum|arXiv (Cornell University)|Mar 15, 2026

Topic Modeling被引用 0

一句话总结

本论文批判LAPDOG在检索增强个性化对话中的评估与数据实践，指出表面相似度指标与基于连贯性、一致性与共识的人工/LLM判断存在偏离，并提出具认知信息的评估方向。

ABSTRACT

In cognitive science and linguistic theory, dialogue is not seen as a chain of independent utterances but rather as a joint activity sustained by coherence, consistency, and shared understanding. However, many systems for open-domain and personalized dialogue use surface-level similarity metrics (e.g., BLEU, ROUGE, F1) as one of their main reporting measures, which fail to capture these deeper aspects of conversational quality. We re-examine a notable retrieval-augmented framework for personalized dialogue, LAPDOG, as a case study for evaluation methodology. Using both human and LLM-based judges, we identify limitations in current evaluation practices, including corrupted dialogue histories, contradictions between retrieved stories and persona, and incoherent response generation. Our results show that human and LLM judgments align closely but diverge from lexical similarity metrics, underscoring the need for cognitively grounded evaluation methods. Broadly, this work charts a path toward more reliable assessment frameworks for retrieval-augmented dialogue systems that better reflect the principles of natural human communication.

研究动机与目标

• 质疑当前评估实践在检索增强型个性化对话中对对话质量的捕捉
• 以LAPDOG为案例研究，识别历史、检索与度量的局限性
• 提出一个结合人类与LLM判断的框架，基于话语与认知理论
• 提出能反映连贯性、参与度与共识的度量方向。

提出的方法

• 在被腐蚀的对话上复现LAPDOG实验以验证基线增益
• 在未被腐蚀的CONVAI2数据上重新训练LAPDOG及基线，并收集人工与LLM评估
• 使用两名人工评审与两名LLM评估者（源自ChatGPT与DeepSeek）对回答进行1–5分打分并对候选进行排序
• 将表面相似度指标（BLEU、ROUGE、F1）与人工/LLM判断进行比较
• 使用Pearson相关性和Williams检验分析人工、LLM与词汇指标之间的相关性

Figure 1: Overview of the LAPDOG retrieval-augmented personalized dialogue framework. The model retrieves external stories (e.g., from ROCStory) based on persona and dialogue history using a dual-encoder retriever, integrates them to a generator, and evaluates responses with metrics such as BLEU and

实验结果

研究问题

• RQ1 表面词汇指标是否与人工和LLM在连贯性与人格一致性判断上保持一致？
• RQ2 在认知和语言学层面评估时，LAPDOG的检索-历史-连贯性管线有哪些局限？
• RQ3 人类与LLM评估者是否能提供可靠、具认知基础的对话质量评估？
• RQ4 哪些评估框架与检索筛选策略能提升RAG个性化中的连贯性与共识？

主要发现

• 人工与LLM的判断彼此一致，但与BLEU/ROUGE/F1等表面指标存在差异
• 在被腐蚀的历史记录和表面指标下获得的增益并不始终转化为在未腐蚀数据上的更高感知质量
• 检索的故事内容可能与人格信息相矛盾，降低可信度与连贯性
• 被腐蚀的对话历史破坏话语结构，削弱跨轮对话的连贯性
• 词汇重叠指标与人工/LLM判断相关性弱甚至为负，强调需要具认知基础的评估指标
• LLM评估者在该设定中能较好地近似人工判断，显示基于LLM的评估在实践中具有可行性