[Paper Review] User Embedding Model for Personalized Language Prompting

Sumanth Doddapaneni, Krishna Sayana+3 more|arXiv (Cornell University)|Jan 10, 2024

Topic Modeling4 citations

TL;DR

This paper proposes a User Embedding Module (UEM) that compresses long user histories into contextual embeddings, which are used as soft prompts to personalize language model outputs. By co-training the UEM with the language model, it enables efficient, scalable personalization with up to 0.25 F1-point improvement over text-based prompting baselines.

ABSTRACT

Modeling long histories plays a pivotal role in enhancing recommendation systems, allowing to capture user's evolving preferences, resulting in more precise and personalized recommendations. In this study we tackle the challenges of modeling long user histories for preference understanding in natural language. Specifically, we introduce a new User Embedding Module (UEM) that efficiently processes user history in free-form text by compressing and representing them as embeddings, to use them as soft prompts to a LM. Our experiments demonstrate the superior capability of this approach in handling significantly longer histories compared to conventional text based prompting methods, yielding substantial improvements in predictive performance. The main contribution of this research is to demonstrate the ability to bias language models with user signals represented as embeddings.

Motivation & Objective

• To address the challenge of modeling long user histories in personalized language modeling for improved preference understanding.
• To overcome the computational inefficiency of directly concatenating long user histories into input tokens, which incurs O(n²) self-attention cost.
• To enable effective personalization by learning user-specific representations that are contextually aligned with downstream tasks.
• To explore the feasibility of using full user histories—rather than sampled or summarized snippets—as input to large language models.
• To demonstrate that user signal embeddings can be used to bias language models effectively, improving predictive performance in recommendation tasks.

Proposed method

• The User Embedding Module (UEM) processes each item in a user's history (e.g., movie title, genre, rating, description) using SentenceT5 to generate dense embeddings.
• These embeddings are combined into a sequence of length p (number of history items), forming a user history representation U ∈ ℝ^{p×3s}, where s is the SentenceT5 embedding dimension.
• A transformer-based UEM processes the sequence of embeddings and applies a learnable linear projection to align the output dimension with the language model’s hidden size e.
• The resulting user prompt embedding Pr_{UEM}(U) ∈ ℝ^{p×e} is concatenated with a task-specific soft prompt P_e ∈ ℝ^{k×e} and the input query embedding to form the final input to the language model.
• The UEM is jointly trained with the language model to ensure user representations are optimized for downstream task performance.
• The method avoids O(n²) attention cost by replacing raw text history with compact, learned embeddings, enabling efficient processing of long histories.

Figure 1: Overview of our User History Modeling Approach. The user history’s textual features are processed through the User Embedding module and combined with the task prompt and subsequently passed through the language model.

Experimental results

Research questions

• RQ1 Can a learned user embedding module effectively compress long user histories into a form suitable for soft prompting in large language models?
• RQ2 How does the performance of the proposed UEM-based prompting compare to conventional text-based prompting with long user histories?
• RQ3 To what extent can the UEM improve predictive performance in preference understanding tasks compared to baselines using sampled or summarized user history?
• RQ4 How does the size of the UEM (e.g., number of layers) affect model performance and efficiency?
• RQ5 Can the UEM be extended to support multimodal user signals through domain-specific embeddings?

Key findings

• The proposed UEM-based prompting method achieves up to 0.21 and 0.25 F1-point improvements over text-based prompting baselines on the movie genre prediction task.
• The UEM enables efficient processing of significantly longer user histories than traditional text-based methods, reducing computational cost from O(n²) to linear in sequence length.
• Increasing the number of UEM layers from 1 to 3 improved F1 scores from 0.346 to 0.396 for the base model and from 0.347 to 0.381 for the large model, indicating a positive correlation between UEM size and performance.
• The method outperforms baseline models such as FlanT5 and LongT5, with the best-performing UEM-enhanced model achieving a test F1 of 0.396 on the base model and 0.381 on the large model.
• The UEM’s performance is sensitive to the choice of embedding model, with SentenceT5-based representations showing strong baseline performance that could be further improved with domain-specific embeddings.
• The approach is generalizable and can be extended to multimodal inputs by using modality-specific embeddings, with future work planned in this direction.

Figure 2: Comparison of model performance with increasing User History.