[Paper Review] Self-supervised Learning for Deep Models in Recommendations.

Tiansheng Yao, Xinyang Yi+8 more|arXiv (Cornell University)|Jul 25, 2020

Topic ModelingComputer Science38 references33 citations

TL;DR

This paper proposes a multi-task self-supervised learning (SSL) framework for deep neural recommenders, introducing two novel tasks—Feature Masking (FM) and Feature Dropout (FD)—to improve embedding quality in large-vocab categorical feature spaces. Evaluated on two large-scale datasets (500M and 1B examples), the framework significantly outperforms supervised-only learning and state-of-the-art regularization methods, especially under low supervision.

ABSTRACT

Large scale neural recommender models play a critical role in modern search and recommendation systems. To model large-vocab categorical features, typical recommender models learn a joint embedding space for both queries and items. With millions to billions of items to choose from, the quality of learned embedding representations is crucial to provide high quality recommendations to users with various interests. Inspired by the recent success in self-supervised representation learning (SSL) research in both computer vision and natural language understanding, we propose a multi-task self-supervised learning framework for neural models in recommendations. Furthermore, we propose two self-supervised tasks applicable to models with categorical features within the proposed framework: (i) Feature Masking (FM) and (ii) Feature Dropout (FD). We evaluate our framework using two large-scale datasets with ~500M and 1B training examples respectively. Our results demonstrate that the proposed framework outperforms learning with the supervision task only and other state-of-the-art regularization techniques in the context of retrieval. The SSL framework shows larger improvement with less supervision compared to the counterparts.

Motivation & Objective

• To improve the quality of learned embedding representations in deep neural recommenders for large-vocabulary categorical features.
• To address the challenge of limited supervision in large-scale recommendation systems by leveraging self-supervised learning (SSL).
• To design effective SSL tasks tailored to categorical features in recommendation models.
• To demonstrate that SSL can achieve superior performance compared to supervised learning and existing regularization techniques, particularly when labeled data is scarce.

Proposed method

• Proposes a multi-task SSL framework that jointly optimizes the supervised recommendation task with self-supervised tasks.
• Introduces Feature Masking (FM), where random categorical features are masked and the model predicts the masked features from context.
• Introduces Feature Dropout (FD), where random categorical features are dropped during training and the model reconstructs them from the remaining features.
• Uses contrastive learning principles to align positive and negative views of the same input under different masking or dropout patterns.
• Applies the framework to deep neural recommenders that jointly embed queries and items in a shared space.
• Trains the model end-to-end using both supervision and the two proposed self-supervised tasks to improve generalization and representation quality.

Experimental results

Research questions

• RQ1 Can self-supervised learning improve the quality of learned embeddings in deep recommendation models with large-vocabulary categorical features?
• RQ2 How do the proposed self-supervised tasks—Feature Masking and Feature Dropout—contribute to better representation learning in recommendation systems?
• RQ3 Does the proposed SSL framework achieve better performance than supervised learning alone, especially when labeled data is limited?
• RQ4 How does the framework compare to state-of-the-art regularization techniques in retrieval performance?

Key findings

• The proposed SSL framework outperforms learning with only the supervision task on both large-scale datasets with 500M and 1B training examples.
• The framework achieves larger improvements under low supervision settings compared to baseline methods, demonstrating stronger data efficiency.
• Feature Masking and Feature Dropout effectively enhance representation learning by encouraging robustness to missing or corrupted features.
• The multi-task SSL framework consistently improves retrieval performance across both datasets, indicating strong generalization capability.
• The gains from SSL are more pronounced when supervision is scarce, highlighting the framework's potential for data-scarce recommendation scenarios.