[Paper Review] ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation

Yu Wang, Hengrui Zhang+3 more|arXiv (Cornell University)|Aug 27, 2022

Recommender Systems and Techniques4 citations

TL;DR

ContrastVAE proposes a contrastive variational autoencoder for sequential recommendation that integrates contrastive learning into VAEs to address data sparsity, uncertainty, and long-tail item challenges. By introducing a two-view training objective (ContrastELBO) and novel augmentation strategies—model and variational augmentation—it improves representation learning and robustness, achieving state-of-the-art performance on four benchmark datasets with up to 7.6% improvement in Recall@40.

ABSTRACT

Aiming at exploiting the rich information in user behaviour sequences, sequential recommendation has been widely adopted in real-world recommender systems. However, current methods suffer from the following issues: 1) sparsity of user-item interactions, 2) uncertainty of sequential records, 3) long-tail items. In this paper, we propose to incorporate contrastive learning into the framework of Variational AutoEncoders to address these challenges simultaneously. Firstly, we introduce ContrastELBO, a novel training objective that extends the conventional single-view ELBO to two-view case and theoretically builds a connection between VAE and contrastive learning from a two-view perspective. Then we propose Contrastive Variational AutoEncoder (ContrastVAE in short), a two-branched VAE model with contrastive regularization as an embodiment of ContrastELBO for sequential recommendation. We further introduce two simple yet effective augmentation strategies named model augmentation and variational augmentation to create a second view of a sequence and thus making contrastive learning possible. Experiments on four benchmark datasets demonstrate the effectiveness of ContrastVAE and the proposed augmentation methods. Codes are available at https://github.com/YuWang-1024/ContrastVAE

Motivation & Objective

• To address the limitations of existing sequential recommendation models, including data sparsity, uncertainty in sequential patterns, and poor generalization to long-tail items.
• To mitigate posterior collapse and point-estimation issues in standard VAEs by integrating contrastive learning into the variational autoencoding framework.
• To develop effective data augmentation strategies that generate semantically meaningful second views of user sequences for contrastive learning.
• To establish a theoretical connection between VAEs and contrastive learning through a novel two-view evidence lower bound (ContrastELBO).
• To improve model robustness against noisy or corrupted input sequences commonly found in real-world user behavior data.

Proposed method

• Proposes ContrastELBO, a two-view extension of the standard ELBO that unifies VAE training with contrastive learning by maximizing mutual information between two augmented views of the same sequence.
• Designs ContrastVAE, a two-branch VAE architecture where each branch processes a different augmented view of the same user behavior sequence, sharing parameters across branches.
• Introduces model augmentation, which generates a second view by applying a perturbed version of the same model to the same input sequence, preserving semantic consistency.
• Introduces variational augmentation, which generates a second view by sampling from the variational posterior distribution of the latent code, introducing stochasticity while maintaining semantic relevance.
• Optimizes the model using a joint loss combining reconstruction loss, KL divergence, and mutual information maximization between the two views to improve representation diversity and robustness.
• Employs a learnable weighting hyperparameter for the mutual information loss to balance next-item prediction and contrastive learning objectives.

Experimental results

Research questions

• RQ1 Can contrastive learning effectively mitigate posterior collapse and variance vanishing in VAE-based sequential recommendation models?
• RQ2 How do model and variational augmentation strategies improve the quality and consistency of positive sample pairs in contrastive learning for sequential data?
• RQ3 To what extent does ContrastVAE improve performance on long-tail items compared to standard VAEs and self-attention baselines?
• RQ4 How robust is ContrastVAE to noisy or corrupted input sequences, such as those with missing or replaced items?
• RQ5 What is the optimal trade-off between contrastive learning and next-item prediction objectives in terms of hyperparameter tuning?

Key findings

• ContrastVAE achieves state-of-the-art performance on four benchmark datasets (Beauty, Sports, Office, and Toys), with up to a 7.6% improvement in Recall@40 compared to baseline models.
• The model significantly reduces posterior collapse, as evidenced by higher KL divergence values between posterior and prior distributions, especially for low-frequency and long-tail items.
• ContrastVAE maintains higher variance in latent representations for rare items, preventing point-estimation collapse and enhancing robustness to input perturbations.
• The model demonstrates superior robustness under both random deletion and replacement of items in sequences, outperforming ACVAE, S3Rec, and SASRec across all corruption ratios.
• Hyperparameter sensitivity analysis shows that a moderate weight on the mutual information loss yields optimal performance, with performance degrading if the contrastive loss is overemphasized.
• The ablation study confirms that both model and variational augmentation contribute meaningfully to performance gains, with the combination yielding the best results.