[论文解读] MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning

Zayne Sprague, Xi Ye+3 more|arXiv (Cornell University)|Oct 24, 2023

Topic Modeling被引用 5

一句话总结

MuSR 提出了一种新颖的基准，通过神经符号的合成到自然语言转换流程，利用自然语言叙述来评估大语言模型中的多步软性推理。该数据集通过复杂、富含常识的故事情节（例如1000字的谋杀悬案）挑战GPT-4等模型，揭示了尽管采用了先进的提示工程技术，其链式思维推理仍存在持续性缺陷。

ABSTRACT

While large language models (LLMs) equipped with techniques like chain-of-thought prompting have demonstrated impressive capabilities, they still fall short in their ability to reason robustly in complex settings. However, evaluating LLM reasoning is challenging because system capabilities continue to grow while benchmark datasets for tasks like logical deduction have remained static. We introduce MuSR, a dataset for evaluating language models on multistep soft reasoning tasks specified in a natural language narrative. This dataset has two crucial features. First, it is created through a novel neurosymbolic synthetic-to-natural generation algorithm, enabling the construction of complex reasoning instances that challenge GPT-4 (e.g., murder mysteries roughly 1000 words in length) and which can be scaled further as more capable LLMs are released. Second, our dataset instances are free text narratives corresponding to real-world domains of reasoning; this makes it simultaneously much more challenging than other synthetically-crafted benchmarks while remaining realistic and tractable for human annotators to solve with high accuracy. We evaluate a range of LLMs and prompting techniques on this dataset and characterize the gaps that remain for techniques like chain-of-thought to perform robust reasoning.

研究动机与目标

• 为解决缺乏将自然语言叙述与多步常识推理相结合的真实复杂推理基准的问题。
• 开发一种可扩展的神经符号数据集生成方法，生成事实一致、人类可解的叙述，其基础为推理树。
• 评估链式思维提示和神经符号方法在复杂现实推理任务中的鲁棒性。
• 识别尽管采用了先进提示策略，当前最先进大语言模型中仍存在的持续性推理缺陷。

提出的方法

• 采用神经符号的合成到自然语言生成流程，首先构建推理树，编码来自特定领域事实的多步推理。
• 利用大语言模型递归扩展推理树，以编码推导正确答案所必需的常识性推理。
• 使用GPT-4以迭代块的形式生成叙述，每一步均验证事实一致性和记忆召回。
• 该过程确保叙述的真实性与逻辑连贯性，同时保留核心事实和推理路径。
• 数据集构建涵盖三个领域：谋杀悬案、物品放置和团队分配，每个领域均需深层次的常识与系统2型推理。
• 评估包括零样本和少样本的链式思维提示，以及神经符号基线方法，在跨领域的750个实例上进行。

Figure 1: Dataset construction process for MuSR. First, we generate gold facts that are used to deduce the correct answer (the murderer in this case). Then, using an LLM, we create a reasoning tree leading to those deductions from facts in a story combined with commonsense. Finally, we iteratively g

实验结果

研究问题

• RQ1 大语言模型（如GPT-4）能否在没有显式中间监督的情况下，从长篇自然语言叙述中可靠地恢复推理路径？
• RQ2 链式思维提示与神经符号方法在嵌入复杂叙述中的多步常识推理任务中表现如何？
• RQ3 当前大语言模型在需要持续注意力与多条事实整合的叙述中，其推理能力失败的程度有多大？
• RQ4 随着模型的发展，神经符号生成方法在生成日益复杂的推理叙述方面具有多大可扩展性？

主要发现

• GPT-4 无论采用何种提示策略，均无法从最终叙述中恢复正确的推理路径，表明自然语言叙述中推理并未被稳健编码。
• 链式思维提示在MuSR上的改进有限，揭示了即使在最先进模型中，推理缺陷依然存在。
• 神经符号生成流程成功生成了长达1000字、事实一致且人类可解的叙述，实现了可扩展基准的构建。
• 该数据集无法被基于规则的系统解决，证实了其复杂性与评估高级推理能力的适用性。
• 人类标注者以高准确率解决了MuSR实例，验证了数据集的真实性与可解性。
• MuSR通过结合自然语言、多步推理、常识和中间推理结构，优于现有基准。

Figure 2: Partial reasoning trees showing gold facts $F$ , story facts $S(T)$ , and commonsense facts $C(T)$ for each of our three domains. Dotted lines indicate incomplete trees. Each deduction sampled from an LLM will yield two scenario facts and one commonsense fact in our setup.