[Paper Review] SCM: Enhancing Large Language Model with Self-Controlled Memory Framework

Bing Wang, Xinnian Liang+7 more|arXiv (Cornell University)|Apr 26, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Self-Controlled Memory (SCM), a plug-and-play framework that enables large language models (LLMs) to process ultra-long inputs without model fine-tuning or architectural changes. By integrating a memory controller, memory stream, and LLM-based agent, SCM dynamically selects and injects relevant historical information, significantly improving long-context reasoning and summarization performance over baselines like ChatGPT.

ABSTRACT

Large Language Models (LLMs) are constrained by their inability to process lengthy inputs, resulting in the loss of critical historical information. To address this limitation, in this paper, we propose the Self-Controlled Memory (SCM) framework to enhance the ability of LLMs to maintain long-term memory and recall relevant information. Our SCM framework comprises three key components: an LLM-based agent serving as the backbone of the framework, a memory stream storing agent memories, and a memory controller updating memories and determining when and how to utilize memories from memory stream. Additionally, the proposed SCM is able to process ultra-long texts without any modification or fine-tuning, which can integrate with any instruction following LLMs in a plug-and-play paradigm. Furthermore, we annotate a dataset to evaluate the effectiveness of SCM for handling lengthy inputs. The annotated dataset covers three tasks: long-term dialogues, book summarization, and meeting summarization. Experimental results demonstrate that our method achieves better retrieval recall and generates more informative responses compared to competitive baselines in long-term dialogues. (https://github.com/wbbeyourself/SCM4LLMs)

Motivation & Objective

• To address the limitation of LLMs in handling ultra-long inputs due to context window constraints and attention computation costs.
• To enable LLMs to maintain and retrieve relevant long-term memory without forgetting critical historical information.
• To develop a plug-and-play framework that requires no model fine-tuning or architectural modifications.
• To evaluate the framework on long-context tasks such as dialogue, book summarization, and meeting summarization.
• To create a new annotated dataset for benchmarking long-context LLM performance.

Proposed method

• The SCM framework consists of three components: an LLM-based agent, a memory stream for storing long-term memories, and a memory controller for dynamic memory management.
• Input text is segmented and processed iteratively, with the memory controller deciding when and how to inject relevant memories from the memory stream into the LLM input.
• The framework uses two memory types: activation memory (long-term) and flash memory (short-term), enabling retention of historical context across segments.
• Memory retrieval and reorganization are guided by the memory controller, which filters out irrelevant or noisy information to reduce interference.
• The system employs a six-step workflow: input acquisition, memory activation, memory retrieval, memory reorganization, input fusion, and response generation.
• The framework supports hierarchical and iterative summarization by propagating relevant memories across document segments to maintain coherence.

Experimental results

Research questions

• RQ1 Can a plug-and-play framework extend the effective context length of LLMs without fine-tuning or model modification?
• RQ2 How effectively can the memory controller select and inject relevant historical information to improve long-context understanding?
• RQ3 Does the SCM framework outperform existing methods in long dialogue and document summarization tasks?
• RQ4 To what extent does the memory controller reduce the impact of historical noise in long inputs?
• RQ5 Can the framework maintain coherence across hierarchical summarization of ultra-long documents?

Key findings

• The SCM framework enables non-dialogue-optimized LLMs like text-davinci-003 to achieve multi-turn dialogue capabilities comparable to ChatGPT, even with inputs exceeding 200,000 tokens.
• In long document summarization, SCM-based models outperformed baselines in both coherence and coverage, with significant improvements in summarization quality on ultra-long inputs (up to 2 million tokens).
• The memory controller effectively reduced noise by selectively injecting only necessary memories, improving retrieval recall and response informativeness in long dialogues.
• The framework achieved superior performance on all three evaluation tasks—long-term dialogue, book summarization, and meeting summarization—without any model fine-tuning.
• The annotated dataset, covering 20k to 2 million tokens per instance, demonstrates the framework’s capability to handle inputs far beyond standard LLM context windows.
• The plug-and-play nature of SCM allows seamless integration with any instruction-following LLM, enhancing scalability and deployment potential.