Skip to main content
QUICK REVIEW

[论文解读] MemoryBank: Enhancing Large Language Models with Long-Term Memory

Wanjun Zhong, Lianghong Guo|arXiv (Cornell University)|May 17, 2023
AI in Service InteractionsComputer Science被引用 15
一句话总结

MemoryBank 引入了一个面向 LLM 的长期记忆机制,能够存储、检索和更新记忆,灵感来自艾宾浩斯遗忘曲线。其通过 SiliconFriend——一个能回忆过去交互并在开源与闭源模型中适应用户个性 的AI伴侣——来演示。

ABSTRACT

Revolutionary advancements in Large Language Models have drastically reshaped our interactions with artificial intelligence systems. Despite this, a notable hindrance remains-the deficiency of a long-term memory mechanism within these models. This shortfall becomes increasingly evident in situations demanding sustained interaction, such as personal companion systems and psychological counseling. Therefore, we propose MemoryBank, a novel memory mechanism tailored for LLMs. MemoryBank enables the models to summon relevant memories, continually evolve through continuous memory updates, comprehend, and adapt to a user personality by synthesizing information from past interactions. To mimic anthropomorphic behaviors and selectively preserve memory, MemoryBank incorporates a memory updating mechanism, inspired by the Ebbinghaus Forgetting Curve theory, which permits the AI to forget and reinforce memory based on time elapsed and the relative significance of the memory, thereby offering a human-like memory mechanism. MemoryBank is versatile in accommodating both closed-source models like ChatGPT and open-source models like ChatGLM. We exemplify application of MemoryBank through the creation of an LLM-based chatbot named SiliconFriend in a long-term AI Companion scenario. Further tuned with psychological dialogs, SiliconFriend displays heightened empathy in its interactions. Experiment involves both qualitative analysis with real-world user dialogs and quantitative analysis with simulated dialogs. In the latter, ChatGPT acts as users with diverse characteristics and generates long-term dialog contexts covering a wide array of topics. The results of our analysis reveal that SiliconFriend, equipped with MemoryBank, exhibits a strong capability for long-term companionship as it can provide emphatic response, recall relevant memories and understand user personality.

研究动机与目标

  • 解决当前 LLM 缺乏长期记忆的问题并改善持续交互,如 AI 伴侣和个人助理。
  • 提供一个记忆增强框架,存储、检索、更新记忆,并构建用户画像。
  • 证明在开源与闭源 LLMs 以及双语能力(英语/中文)上的广泛通用性。
  • 展示一个实际实现(SiliconFriend),结合心理对话数据进行微调,以提升共情和记忆回忆。

提出的方法

  • 提出 MemoryBank,包含记忆存储、记忆检索与记忆更新。
  • 使用基于 FAISS 的双塔密集检索设置,以高效从编码的记忆项中检索记忆。
  • 通过受艾宾浩斯遗忘曲线启发的模型记忆更新,其中记忆强度 S 是离散的,记忆衰退 t 在回忆时重置。
  • 在分层记忆结构中存储详细的多轮对话、每日事件摘要以及演变中的用户个性。
  • 利用提示生成每日/全局事件摘要和个性洞察,形成用户画像。
  • 展示在开源和闭源 LLMs 上与 SiliconFriend 的集成,使用 LoRA 对开源模型进行参数高效微调。

实验结果

研究问题

  • RQ1如何赋予 LLMs 持久记忆以支持长期交互?
  • RQ2一个记忆存储/检索/更新流程是否能提升 AI 伴侣的回忆、连贯性和个性化?
  • RQ3MemoryBank 是否兼容开源和闭源 LLMs 以及双语环境?
  • RQ4用心理对话数据进行微调是否能提升同理心和支持性回答?

主要发现

  • 搭载 MemoryBank 的 SiliconFriend 展现出强大的长期陪伴能力,包括记忆回忆和同理回应。
  • 跨各种 SiliconFriend 变体的记忆检索准确性和连贯性提升,在英语与中文语境中 ChatGPT 的结果最佳。
  • 在对 10 天的 15 位虚拟用户的模拟对话的定量分析中,显示对过去建议和事件的回忆(如堆排序示例)在记忆探测中发生。
  • SiliconFriend 的变体在用英语和中文数据评估时显示出较高的检索准确性和强的模型排序,语言相关的表现差异。
  • 该框架可与开源(ChatGLM、BELLE)和闭源(ChatGPT)模型一起工作,体现普适性。

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。