[Paper Review] Exploiting Behavioral Consistence for Universal User Representation

Jie Gu, Feng Wang+5 more|arXiv (Cornell University)|Dec 11, 2020

Recommender Systems and Techniques21 references4 citations

TL;DR

This paper proposes Self-supervised User Modeling Network (SUMN), a self-supervised framework that learns universal user representations by leveraging behavioral consistency across user interactions. It introduces a behavioral consistency loss to preserve latent user factors and a multi-hop aggregation layer to enhance representation capacity, achieving state-of-the-art performance on downstream tasks like preference prediction and user profiling without task-specific fine-tuning.

ABSTRACT

User modeling is critical for developing personalized services in industry. A common way for user modeling is to learn user representations that can be distinguished by their interests or preferences. In this work, we focus on developing universal user representation model. The obtained universal representations are expected to contain rich information, and be applicable to various downstream applications without further modifications (e.g., user preference prediction and user profiling). Accordingly, we can be free from the heavy work of training task-specific models for every downstream task as in previous works. In specific, we propose Self-supervised User Modeling Network (SUMN) to encode behavior data into the universal representation. It includes two key components. The first one is a new learning objective, which guides the model to fully identify and preserve valuable user information under a self-supervised learning framework. The other one is a multi-hop aggregation layer, which benefits the model capacity in aggregating diverse behaviors. Extensive experiments on benchmark datasets show that our approach can outperform state-of-the-art unsupervised representation methods, and even compete with supervised ones.

Motivation & Objective

• To develop a universal user representation model that generalizes across diverse downstream tasks without task-specific retraining.
• To address the limitation of task-specific models in generalization and high resource cost.
• To design a self-supervised learning objective that captures long-term user behavioral consistency.
• To improve representation quality for diverse, non-coherent user behaviors through enhanced aggregation.
• To enable transferable, informative user embeddings applicable to preference prediction and user profiling.

Proposed method

• Proposes a novel behavioral consistency loss that predicts target behavior frequencies from input behavior sequences, encouraging the model to learn stable user factors.
• Introduces a multi-hop aggregation layer using shared-parameter attention mechanisms to iteratively refine user representations across multiple hops.
• Employs a dual-branch input structure: one for input behavior sequences and another for target behavior sequences, both encoded into a shared universal representation space.
• Uses self-supervised pre-training on large-scale unlabeled user logs, eliminating the need for labeled data in representation learning.
• Applies mean pooling and max pooling as ablation baselines to evaluate the effectiveness of the multi-hop aggregation design.
• Employs PCA for visualization of learned representations to demonstrate clustering by user preference.

Experimental results

Research questions

• RQ1 Can a self-supervised objective based on behavioral consistency improve universal user representation learning?
• RQ2 How does multi-hop attention aggregation compare to simple pooling in summarizing diverse user behaviors?
• RQ3 To what extent can universal representations learned without labels generalize to downstream tasks like preference prediction?
• RQ4 Does the absence of explicit sequence modeling affect performance, and can transformers improve results?
• RQ5 How well do the learned representations cluster users by latent preferences without access to preference labels?

Key findings

• SUMN achieves 64.47% accuracy on user age prediction and 72.17% on baby-age prediction, outperforming all baselines including supervised methods on the industrial dataset.
• The multi-hop aggregation layer significantly improves performance over mean and max pooling, especially for users with high behavioral diversity.
• SUMN with behavioral consistency loss consistently outperforms SUMN-AE and SUMN-ID, demonstrating the effectiveness of the proposed objective.
• Replacing the multi-hop layer with a transformer yields less than 1% performance gain, indicating limited benefit from explicit sequence modeling for long-term user factors.
• Visualization via PCA shows that users with the same category preference are embedded more closely, confirming the model learns semantically meaningful representations.
• SUMN achieves competitive performance against supervised methods despite being pre-trained on unlabeled data, proving the viability of self-supervised universal user modeling.