[论文解读] RECKONING: Reasoning through Dynamic Knowledge Encoding

Zeming Chen, Gail Garfinkel Weiss+3 more|arXiv (Cornell University)|May 10, 2023

Topic Modeling被引用 5

一句话总结

本文提出 Reckoning，一种双层优化方法，通过在推理过程中利用梯度更新将动态上下文知识编码到模型参数中，使基于 Transformer 的语言模型能够稳健地进行推理。通过学习快速调整权重以记忆相关事实，Reckoning 在推理性能上相比上下文内推理最高提升 4.5%，尤其在存在干扰项或长链推理场景下表现更优。

ABSTRACT

Recent studies on transformer-based language models show that they can answer questions by reasoning over knowledge provided as part of the context (i.e., in-context reasoning). However, since the available knowledge is often not filtered for a particular question, in-context reasoning can be sensitive to distractor facts, additional content that is irrelevant to a question but that may be relevant for a different question (i.e., not necessarily random noise). In these situations, the model fails to distinguish the knowledge that is necessary to answer the question, leading to spurious reasoning and degraded performance. This reasoning failure contrasts with the model's apparent ability to distinguish its contextual knowledge from all the knowledge it has memorized during pre-training. Following this observation, we propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model's parameters before presenting it with a question. Our method, RECKONING, is a bi-level learning algorithm that teaches language models to reason by updating their parametric knowledge through back-propagation, allowing them to then answer questions using the updated parameters. During training, the inner loop rapidly adapts a copy of the model weights to encode contextual knowledge into its parameters. In the outer loop, the model learns to use the updated weights to reproduce and answer reasoning questions about the memorized knowledge. Our experiments on two multi-hop reasoning datasets show that RECKONING's performance improves over the in-context reasoning baseline (by up to 4.5%). We also find that compared to in-context reasoning, RECKONING generalizes better to longer reasoning chains unseen during training, is more robust to distractors in the context, and is more computationally efficient when multiple questions are asked about the same knowledge.

研究动机与目标

• 解决上下文内推理对上下文知识中干扰事实的敏感性问题。
• 通过教会模型区分相关与无关的上下文事实，提升推理鲁棒性。
• 使语言模型能够在推理过程中将外部知识编码到其参数中，模拟参数化知识。
• 开发一种元学习框架，以优化模型参数，实现有效的动态知识编码与推理。
• 提升模型在更长推理链上的泛化能力，并在针对同一知识回答多个问题时降低计算成本。

提出的方法

• Reckoning 使用双层优化框架：内层循环通过语言建模损失执行快速权重更新，将上下文知识编码到模型参数中。
• 外层循环优化初始元参数，使经微调后的模型能够基于编码的知识正确回答推理问题。
• 内层循环更新采用按层和步骤动态调整的学习率，通过梯度下降学习，以提升适应效率。
• 该方法在多跳推理数据集上进行训练，联合优化知识记忆与下游推理准确率。
• 在推理阶段，模型在回答问题前执行若干梯度步骤，将新事实编码到其权重中。
• 该框架通过联合语言建模损失与推理准确率损失的总损失，实现端到端训练。

Figure 1 : Our algorithm, Reckoning , solves reasoning problems by encoding external contextual knowledge into a model’s parameters through gradient updates. At inference time, Reckoning performs a few parameter updates using the gradients of a language modeling loss to encode the relevant facts. Th

实验结果

研究问题

• RQ1 能否通过将知识编码到模型参数中而非依赖上下文内提示，来训练语言模型实现更稳健的上下文知识推理？
• RQ2 与标准的上下文内推理相比，推理过程中动态参数自适应是否能提升性能，尤其在存在干扰项的情况下？
• RQ3 当知识以参数化方式编码而非上下文化方式时，模型在更长推理链上的泛化能力是否更强？
• RQ4 内层循环中使用动态学习率如何影响模型的整体推理性能？
• RQ5 当针对同一知识提出多个问题时，所提出方法是否比上下文内推理更具计算效率？

主要发现

• 在三个多跳推理数据集上，Reckoning 相较于上下文内推理基线，推理准确率最高提升 4.5%。
• 在 CLUTRR-SG 数据集上，Reckoning 在 4 跳推理链上达到 98.3% 的准确率，在 6 跳链上达到 94.8%，显著优于零样本 GPT-3.5。
• 在存在干扰项的情况下，Reckoning 仍保持高性能（例如，在含干扰项的 5 跳 ProofWriter 上达到 84.0%），而 GPT-3.5 下降至 45.3%。
• 与固定学习率相比，内层循环中使用动态学习率使性能下降减少 34.2%，在更长链上提升更大（4 跳链下降减少 45.5%）。
• Reckoning 在未见推理链长度上泛化能力更强，且在针对同一知识回答多个问题时计算效率更高。
• Reckoning 在对抗性设置（含干扰项）下，优于微调后的上下文内推理以及 GPT-3.5 的零样本/少样本提示，表现更优。

Figure 2 : The two-stage training process of Reckoning with an inner and outer loop.