[Paper Review] Hello Again! LLM-powered Personalized Agent for Long-term Dialogue

Hao Li, Chenghao Yang+4 more|arXiv (Cornell University)|Jun 9, 2024

Speech and dialogue systems4 citations

TL;DR

This paper introduces LD-Agent, a model-agnostic, LLM-powered framework for long-term, personalized dialogue that integrates event memory and persona modeling through three modular components: event perception, persona extraction, and response generation. It achieves state-of-the-art performance on MSC and Conversation Chronicles benchmarks, demonstrating strong zero-shot generalization and cross-domain transferability across diverse models and tasks.

ABSTRACT

Open-domain dialogue systems have seen remarkable advancements with the development of large language models (LLMs). Nonetheless, most existing dialogue systems predominantly focus on brief single-session interactions, neglecting the real-world demands for long-term companionship and personalized interactions with chatbots. Crucial to addressing this real-world need are event summary and persona management, which enable reasoning for appropriate long-term dialogue responses. Recent progress in the human-like cognitive and reasoning capabilities of LLMs suggests that LLM-based agents could significantly enhance automated perception, decision-making, and problem-solving. In response to this potential, we introduce a model-agnostic framework, the Long-term Dialogue Agent (LD-Agent), which incorporates three independently tunable modules dedicated to event perception, persona extraction, and response generation. For the event memory module, long and short-term memory banks are employed to separately focus on historical and ongoing sessions, while a topic-based retrieval mechanism is introduced to enhance the accuracy of memory retrieval. Furthermore, the persona module conducts dynamic persona modeling for both users and agents. The integration of retrieved memories and extracted personas is subsequently fed into the generator to induce appropriate responses. The effectiveness, generality, and cross-domain capabilities of LD-Agent are empirically demonstrated across various illustrative benchmarks, models, and tasks. The code is released at https://github.com/leolee99/LD-Agent.

Motivation & Objective

• To address the gap in long-term, personalized dialogue systems that currently focus on short, single-session interactions.
• To enable sustained, coherent, and contextually consistent conversations by maintaining long-term event memory and dynamic persona modeling.
• To develop a model-agnostic framework that generalizes across diverse LLM architectures and real-world dialogue domains.
• To enhance response generation through integrated retrieval of historical events and dynamically updated user-agent personas.
• To evaluate the framework’s effectiveness, generality, and transferability across multiple benchmarks, models, and dialogue tasks.

Proposed method

• The framework consists of three independently tunable modules: event memory perception, persona extraction, and response generation.
• Event memory uses separate long-term and short-term memory banks, with topic-based retrieval to improve accuracy in recalling past events.
• The persona extraction module employs a disentangled, tunable mechanism to model both user and agent personas dynamically and store them in a long-term bank.
• Retrieved memories and extracted personas are jointly encoded and fed into the response generator to produce contextually appropriate, personalized responses.
• The framework is designed to be model-agnostic, enabling deployment across various LLMs and fine-tuning on different datasets without architectural re-design.
• Cross-domain evaluation is performed by tuning on one dataset (e.g., CC) and testing on another (e.g., MSC), and cross-task evaluation is conducted on multiparty dialogue using the Ubuntu IRC dataset.

Experimental results

Research questions

• RQ1 Can a model-agnostic framework effectively maintain long-term coherence in open-domain dialogue by integrating event memory and persona modeling?
• RQ2 How well does LD-Agent generalize across different dialogue domains, especially when fine-tuned on one dataset and tested on another with a domain gap?
• RQ3 To what extent can LD-Agent be transferred to different dialogue tasks, such as multiparty conversation, without architectural modification?
• RQ4 Does the integration of retrieved memories and dynamically updated personas significantly improve response quality and consistency in long-term interactions?
• RQ5 How does LD-Agent perform in zero-shot and few-shot settings compared to existing methods across diverse benchmarks?

Key findings

• On the MSC benchmark, LD-Agent achieved a R-L score of 23.31 in the MSC-tuning LDA setting, significantly outperforming the previous SOTA method (19.94) and zero-shot baselines (16.63).
• On the Conversation Chronicles (CC) benchmark, LD-Agent reached a R-L score of 39.67 in the CC-tuning LDA setup, surpassing the previous SOTA (37.61) and zero-shot models (22.10).
• In cross-domain evaluation, models fine-tuned on one dataset (e.g., CC) still performed well on the other (e.g., MSC), with only a slight drop in performance compared to in-domain tuning, demonstrating strong domain generalization.
• In multiparty dialogue on the Ubuntu IRC dataset, LD-Agent with BART backbone achieved a R-L score of 12.28, outperforming HeterMPC BART (11.20) and other baselines, confirming strong task transferability.
• The framework showed consistent improvements across all metrics (BL-2, BL-3, R-L) across both benchmarks, indicating robust and stable performance gains.
• The zero-shot performance of LD-Agent was consistently better than baseline models, highlighting its strong inductive bias and generalization capability even without fine-tuning.