[Paper Review] Pearl: Personalizing Large Language Model Writing Assistants with Generation-Calibrated Retrievers

Sheshera Mysore, Zhuoran Lu+8 more|arXiv (Cornell University)|Nov 15, 2023

Topic Modeling4 citations

TL;DR

Pearl proposes a retrieval-augmented large language model (LLM) writing assistant that personalizes outputs using a generation-calibrated retriever. The retriever selects user-specific historical documents to augment prompts, trained via a novel data selection method and a scale-calibrated KL-divergence loss to optimize for personalized text generation. Pearl outperforms strong baselines in both automated and manual evaluations on workplace and Reddit text generation tasks.

ABSTRACT

Powerful large language models have facilitated the development of writing assistants that promise to significantly improve the quality and efficiency of composition and communication. However, a barrier to effective assistance is the lack of personalization in LLM outputs to the author's communication style, specialized knowledge, and values. In this paper, we address this challenge by proposing Pearl, a LLM writing assistant personalized with a retriever that is trained to be generation-calibrated for personalization. Generation calibration ensures that our retriever selects historic user authored documents to augment an LLM prompt such that they are likely to help an LLM generation better adhere to a users' preferences. We propose two key novelties for training such a retriever: (1) A training data selection method that identifies user requests likely to benefit from personalization and documents that provide that benefit; and (2) A scale-calibrating KL-divergence objective that ensures that our retriever scores remain proportional to the downstream generation quality from using the document for personalized generation. In a series of holistic evaluations, we demonstrate the effectiveness of Pearl in generating long-form texts on multiple social media datasets. Finally, we demonstrate how a generation-calibrated retriever can double as a performance predictor -- detecting low quality retrieval, and improving potentially under-performing outputs via revision with LLMs.

Motivation & Objective

• To address the lack of personalization in LLM-generated text, which limits user trust and adoption in writing assistants.
• To improve the relevance and stylistic consistency of LLM outputs by retrieving and incorporating user-specific historical documents.
• To train a retriever that is directly optimized for personalized generation performance, rather than generic retrieval.
• To enable the retriever to serve double duty as a performance predictor for low-quality generations.
• To demonstrate effectiveness in real-world writing tasks such as workplace social media posts and Reddit comments.

Proposed method

• A generation-calibrated retriever is trained to select historic user-authored documents that best personalize LLM responses to a given request.
• A data selection method identifies user requests likely to benefit from personalization and documents that provide such benefit.
• A scale-calibrated KL-divergence loss distills scores from an auxiliary generation model into the retriever, aligning retrieval with personalized generation quality.
• The retriever augments the LLM prompt with selected documents as few-shot examples to guide personalized generation.
• The same retriever scores are used to predict generation quality, enabling selective LLM chaining for low-quality outputs.
• The system is evaluated on enterprise workplace communications and public Reddit comment datasets using both automatic metrics and human evaluation.

Experimental results

Research questions

• RQ1 Can a retrieval-augmented approach effectively personalize LLM-generated text using a user's historical writing?
• RQ2 How does a generation-calibrated retriever compare to standard retrieval baselines in personalized text generation?
• RQ3 Can the retriever’s scores be used to predict generation quality and guide LLM chaining?
• RQ4 To what extent does personalization improve stylistic and factual consistency in generated text?
• RQ5 Does the method generalize across different writing domains, such as workplace communication and social media?

Key findings

• Pearl outperforms strong retrieval-augmentation baselines on both automated metrics and human evaluations for personalized writing tasks.
• The generation-calibrated retriever significantly improves the relevance and stylistic consistency of LLM outputs compared to non-calibrated baselines.
• The retriever’s scores successfully predict low-quality generations, enabling selective use of more complex LLM chaining to improve output quality.
• The method demonstrates strong performance on both enterprise workplace communications and public Reddit comment generation tasks.
• The scale-calibrated KL-divergence loss effectively aligns retrieval with personalized generation objectives, leading to measurable gains in generation quality.
• Manual evaluations confirm that Pearl-generated outputs are preferred over baselines in terms of tone, structure, and information content similarity to reference texts.