[Paper Review] Compress to Impress: Unleashing the Potential of Compressive Memory in Real-World Long-Term Conversations

Nuo Chen, Hongguang Li+3 more|arXiv (Cornell University)|Feb 19, 2024

Creativity in Education and NeurosciencePsychology3 citations

TL;DR

This paper proposes COMEDY, a novel LLM-based dialogue system that replaces traditional retrieval-based memory management with a unified 'One-for-All' framework for compressive memory. By integrating session summaries, user-bot dynamics, and past events into a single, compact memory representation, COMEDY achieves more human-like, contextually coherent long-term conversations without relying on external memory databases or retrieval modules, outperforming existing methods on real-world Chinese dialogue benchmarks.

ABSTRACT

Existing retrieval-based methods have made significant strides in maintaining long-term conversations. However, these approaches face challenges in memory database management and accurate memory retrieval, hindering their efficacy in dynamic, real-world interactions. This study introduces a novel framework, COmpressive Memory-Enhanced Dialogue sYstems (COMEDY), which eschews traditional retrieval modules and memory databases. Instead, COMEDY adopts a "One-for-All" approach, utilizing a single language model to manage memory generation, compression, and response generation. Central to this framework is the concept of compressive memory, which intergrates session-specific summaries, user-bot dynamics, and past events into a concise memory format. To support COMEDY, we curated a large-scale Chinese instruction-tuning dataset, Dolphin, derived from real user-chatbot interactions. Comparative evaluations demonstrate COMEDY's superiority over traditional retrieval-based methods in producing more nuanced and human-like conversational experiences. Our codes are available at https://github.com/nuochenpku/COMEDY.

Motivation & Objective

• To address the limitations of retrieval-based memory systems in long-term dialogue, such as performance unpredictability and memory database management complexity.
• To enable more coherent and human-like long-term conversations by eliminating the need for separate memory retrieval and database storage.
• To develop a unified, end-to-end framework that generates, compresses, and utilizes memory within a single language model.
• To create a realistic, large-scale Chinese instruction-tuning dataset (Dolphin) from real user-chatbot interactions to better reflect real-world conversational dynamics.
• To demonstrate that a single, compressive memory representation can effectively support nuanced, contextually rich, and personalized dialogue over extended interactions.

Proposed method

• COMEDY employs a single large language model to jointly handle memory generation, compression, and response generation, eliminating the need for separate modules.
• It constructs a compressive memory by fusing three components: session-specific summaries, user and bot portraits, and dynamic relationship evolution across interactions.
• The framework avoids external memory databases by reprocessing and condensing all past interactions into a single, concise memory representation.
• A large-scale Chinese instruction-tuning dataset named Dolphin was curated from real user-chatbot interactions to train and evaluate the system.
• The system uses few-shot prompting and instruction tuning to align the LLM with the task of generating and maintaining compressive memory.
• The compressive memory is dynamically updated and integrated into the context during inference, enabling long-horizon coherence.

Experimental results

Research questions

• RQ1 Can a single LLM effectively replace multi-module retrieval-based memory systems in long-term dialogue without external memory storage?
• RQ2 How does a unified 'One-for-All' approach compare to traditional retrieval-based methods in terms of response quality and coherence in real-world conversations?
• RQ3 To what extent can a compressive memory representation—integrating summaries, user profiles, and relationship dynamics—improve long-term dialogue coherence and personalization?
• RQ4 How does the performance of COMEDY vary across diverse, real-world conversational scenarios compared to existing benchmarks?
• RQ5 Can a large-scale, real-world Chinese dialogue dataset (Dolphin) effectively support the training of a robust, long-term dialogue system?

Key findings

• COMEDY outperforms traditional retrieval-based methods in generating more nuanced, contextually coherent, and human-like responses in long-term conversations.
• The absence of a retrieval module and memory database leads to more stable and predictable performance, avoiding the pitfalls of embedding-based retrieval failures.
• The compressive memory representation successfully captures and retains key information across sessions, including user preferences, event histories, and evolving relationship dynamics.
• Evaluations on the Dolphin dataset show that COMEDY generates more personalized and emotionally consistent responses over extended dialogues.
• The framework demonstrates robustness in handling diverse, real-world conversational topics, colloquial expressions, and dynamic user behaviors.
• The ablation study confirms that the integration of user portraits and relationship dynamics significantly enhances response quality and coherence.