[论文解读] CiteAudit: You Cited It, But Did You Read It? A Benchmark for Verifying Scientific References in the LLM Era

Zhengqing Yuan, Kaiwen Shi+4 more|arXiv (Cornell University)|Feb 26, 2026

Scientific Computing and Data Management被引用 0

一句话总结

tldr: CiteAudit introduces a multi-agent framework and a large-scale benchmark to verify the faithfulness and evidence alignment of cited references in scientific writing, addressing hallucinated citations in the LLM era. It shows improved detection accuracy and interpretability over baselines and provides standardized evaluation protocols.

ABSTRACT

Scientific research relies on accurate citation for attribution and integrity, yet large language models (LLMs) introduce a new risk: fabricated references that appear plausible but correspond to no real publications. Such hallucinated citations have already been observed in submissions and accepted papers at major machine learning venues, exposing vulnerabilities in peer review. Meanwhile, rapidly growing reference lists make manual verification impractical, and existing automated tools remain fragile to noisy and heterogeneous citation formats and lack standardized evaluation. We present the first comprehensive benchmark and detection framework for hallucinated citations in scientific writing. Our multi-agent verification pipeline decomposes citation checking into claim extraction, evidence retrieval, passage matching, reasoning, and calibrated judgment to assess whether a cited source truly supports its claim. We construct a large-scale human-validated dataset across domains and define unified metrics for citation faithfulness and evidence alignment. Experiments with state-of-the-art LLMs reveal substantial citation errors and show that our framework significantly outperforms prior methods in both accuracy and interpretability. This work provides the first scalable infrastructure for auditing citations in the LLM era and practical tools to improve the trustworthiness of scientific references.

研究动机与目标

• Motivate the need to combat hallucinated citations in scholarly writing due to LLMs.
• Propose a scalable, multi-agent verification framework to assess citation faithfulness and evidence alignment.
• Create a large, human-validated benchmark spanning diverse domains and citation types.
• Provide unified evaluation protocols and metrics for citation verification.
• Demonstrate improved accuracy and interpretability over baselines in experiments with state-of-the-art LLMs.

提出的方法

• 设计一个五代理管道：Claim Extractor, Retriever, Evidence Matcher, Reasoner, and Judge，由一个计划控制器协调.
• 开发一个结合真实世界引用错误与系统性生成的幻觉引用、并经人工验证的大规模数据集.
• 将验证形式化为一个多阶段证据一致性任务，设定严格的验证标准，要求元数据完全匹配。
• 通过代理实现网络搜索和学术数据库作为外部知识源以为证据提供 grounding。
• 在生成的基准与真实世界测试集上使用标准的引用可信度与判决一致性度量来评估模型。

实验结果

研究问题

• RQ1 一个多代理框架能否在科学手稿中可靠地检测幻觉引用？
• RQ2 证据检索与推理如何影响对不同引用类型的可信度判断？
• RQ3 引入权威学者验证对召回率和精确度有何影响？
• RQ4 在实际应用中生成的基准是否能真实反映现实世界的引用错误模式？

主要发现

• 基准结合了真实世界和合成的幻觉引用，呈现出与实际情况相似的现实错误模式。
• 多代理验证显著提升了准确性和可解释性，相较于单模型基线。
• 学者代理作为最终验证阶段，降低了仅通过网络检查就能通过的顽固性幻觉。
• 在真实世界数据上，所提出的框架在所有评估方法中达到最高的准确性、精确度、召回率和F1。
• 与许多专有的基于LLM的解决方案相比，该方法成本与延迟更低，因为将大量推理限定在计划和最终判定阶段。