[Paper Review] Integrating Summarization and Retrieval for Enhanced Personalization via Large Language Models

Chris Richardson, Yao Zhang+6 more|arXiv (Cornell University)|Oct 30, 2023

Topic Modeling4 citations

TL;DR

This paper proposes a summary-augmented retrieval framework that uses LLM-generated user summaries to enhance personalization in NLP systems, reducing reliance on real-time retrieval. By generating task-aware summaries offline, the method achieves performance on par or better than retrieval-only baselines with 75% less retrieved data, especially benefiting cold-start scenarios and low-latency applications like voice assistants.

ABSTRACT

Personalization, the ability to tailor a system to individual users, is an essential factor in user experience with natural language processing (NLP) systems. With the emergence of Large Language Models (LLMs), a key question is how to leverage these models to better personalize user experiences. To personalize a language model's output, a straightforward approach is to incorporate past user data into the language model prompt, but this approach can result in lengthy inputs exceeding limitations on input length and incurring latency and cost issues. Existing approaches tackle such challenges by selectively extracting relevant user data (i.e. selective retrieval) to construct a prompt for downstream tasks. However, retrieval-based methods are limited by potential information loss, lack of more profound user understanding, and cold-start challenges. To overcome these limitations, we propose a novel summary-augmented approach by extending retrieval-augmented personalization with task-aware user summaries generated by LLMs. The summaries can be generated and stored offline, enabling real-world systems with runtime constraints like voice assistants to leverage the power of LLMs. Experiments show our method with 75% less of retrieved user data is on-par or outperforms retrieval augmentation on most tasks in the LaMP personalization benchmark. We demonstrate that offline summarization via LLMs and runtime retrieval enables better performance for personalization on a range of tasks under practical constraints.

Motivation & Objective

• Address the limitations of retrieval-based personalization, including information loss, shallow user understanding, and cold-start problems.
• Overcome input length and latency constraints in LLMs by offloading user data summarization to offline processing.
• Improve personalization performance in low-data or new-user scenarios by leveraging high-level, task-aware user summaries.
• Enable real-time, low-latency applications like voice assistants to benefit from LLM power through precomputed summaries.
• Demonstrate that summarization can reduce dependency on large-scale retrieval without sacrificing performance on diverse NLP tasks.

Proposed method

• Generate task-aware user summaries offline using large language models (e.g., Vicuna, GPT-3.5) based on user history and task context.
• Store these summaries in a knowledge base for low-latency retrieval during inference.
• Augment retrieval-augmented generation with both retrieved user data and precomputed summaries during inference.
• Use a prompt engineering strategy to guide summary models to focus on relevant aspects (e.g., writing style, preferences) depending on the downstream task.
• Integrate summaries with retrieval at inference time, allowing the model to access both high-level abstractions and specific data points.
• Fine-tune a FlanT5-base model on the LaMP benchmark to evaluate performance across classification and generation tasks with varying retrieval and summary configurations.

Figure 1. Personalization is achieved by combining runtime-retrieved samples with an offline-generated user summary. Given a textual input $x$ that describes a task in natural language, the goal is to generate a personalized output $y$ for users. The retrieval model identifies the most relevant item

Experimental results

Research questions

• RQ1 Can LLM-generated user summaries effectively replace or reduce the need for real-time retrieval in personalized NLP tasks?
• RQ2 To what extent does offline summarization improve performance on low-data or cold-start user scenarios compared to retrieval-only methods?
• RQ3 How does the quality of the summary model (e.g., GPT-3.5 vs. Vicuna) impact downstream personalization performance?
• RQ4 Can a hybrid approach of retrieval and summarization reduce input length and inference cost while maintaining or improving accuracy?
• RQ5 In which tasks does summarization provide a performance advantage even when retrieval is disabled?

Key findings

• The proposed method with GPT-3.5-generated summaries outperformed the retrieval-only baseline with k=4 on LaMP-1 (accuracy: 0.743 vs. 0.709) and LaMP-2 (accuracy: 0.814 vs. 0.807), achieving better results with less data.
• On five out of six tasks in the LaMP benchmark, the method achieved performance on par or better than the retrieval-only baseline while using 75% less retrieved user data.
• For LaMP-1, the method using only offline summaries (k=0) achieved an accuracy of 0.738, surpassing the retrieval-only baseline with k=4 (0.709), demonstrating effectiveness in sparse-data scenarios.
• GPT-3.5-generated summaries consistently outperformed Vicuna-generated summaries, likely due to model size and reasoning quality differences.
• The method achieved statistically significant improvements (p < 0.05) on LaMP-1 and LaMP-2 compared to baselines, indicating robustness across tasks.
• On tasks like LaMP-4 and LaMP-5, the method maintained or slightly improved performance with minimal retrieval, showing that high-quality summaries can substitute for extensive retrieval.