[Paper Review] Wasserstein Autoencoders for Collaborative Filtering

Jingbin Zhong, Xiaofeng Zhang|arXiv (Cornell University)|Sep 15, 2018

Recommender Systems and Techniques23 references3 citations

TL;DR

This paper proposes an adapted Wasserstein autoencoder (aWAE) for collaborative filtering using implicit feedback, enhancing model performance by introducing mutual information loss and L1 regularization to improve latent space quality and sparsity. Evaluated on ML-20M, Netflix, and LASTFM, the aWAE outperforms Mult-VAE, Mult-DAE, CDAE, and Slim across Recall@1, Recall@5, and NDCG@10, demonstrating superior recommendation accuracy.

ABSTRACT

The recommender systems have long been investigated in the literature. Recently, users' implicit feedback like `click' or `browse' are considered to be able to enhance the recommendation performance. Therefore, a number of attempts have been made to resolve this issue. Among them, the variational autoencoders (VAE) approach already achieves a superior performance. However, the distributions of the encoded latent variables overlap a lot which may restrict its recommendation ability. To cope with this challenge, this paper tries to extend the Wasserstein autoencoders (WAE) for collaborative filtering. Particularly, the loss function of the adapted WAE is re-designed by introducing two additional loss terms: (1) the mutual information loss between the distribution of latent variables and the assumed ground truth distribution, and (2) the L1 regularization loss introduced to restrict the encoded latent variables to be sparse. Two different cost functions are designed for measuring the distance between the implicit feedback data and its re-generated version of data. Experiments are valuated on three widely adopted data sets, i.e., ML-20M, Netflix and LASTFM. Both the baseline and the state-of-the-art approaches are chosen for the performance comparison which are Mult-DAE, Mult-VAE, CDAE and Slim. The performance of the proposed approach outperforms the compared methods with respect to evaluation criteria Recall@1, Recall@5 and NDCG@10, and this demonstrates the efficacy of the proposed approach.

Motivation & Objective

• To address the limitation of variational autoencoders (VAEs) in collaborative filtering, where latent variable distributions overlap significantly, reducing recommendation performance.
• To adapt Wasserstein autoencoders (WAEs) to collaborative filtering for the first time, leveraging optimal transport for better latent space distribution alignment.
• To improve model generalization and handle data sparsity by introducing L1 regularization to enforce sparsity in encoded latent variables.
• To enhance reconstruction fidelity by designing two novel cost functions, including a multinomial loss that accounts for non-clicked items.
• To validate the proposed method’s superiority over state-of-the-art baselines on large-scale, real-world recommendation datasets.

Proposed method

• Proposes a novel loss function for aWAE that combines reconstruction loss, mutual information loss between latent and prior distributions, and L1 regularization to promote sparsity.
• Introduces a sample mean mean-variance (SMV) method to approximate the Wasserstein distance between the latent distribution and the prior, replacing the original GAN-style discriminator.
• Employs a modified variational inference learning framework with ADMM to solve the constrained optimization problem, enabling separate learning of deep network parameters and regularization terms.
• Designs two distinct cost functions: one based on mean squared error and another multinomial loss that explicitly models non-clicked items to improve training stability and performance.
• Uses variational inference with reparameterization to optimize the encoder and decoder networks end-to-end, while enforcing distributional constraints via optimal transport.
• Applies ADMM to optimize the L1 regularization term, decoupling the learning of network weights and the regularization parameter.

Experimental results

Research questions

• RQ1 Can a Wasserstein autoencoder be effectively adapted to collaborative filtering with implicit feedback, improving upon existing VAE-based methods?
• RQ2 How does introducing mutual information loss between the latent and prior distributions affect the quality and disentanglement of learned representations?
• RQ3 To what extent does L1 regularization on latent variables improve sparsity and recommendation performance on sparse, real-world datasets?
• RQ4 Does a multinomial cost function that models non-clicked items lead to faster convergence and better performance than standard reconstruction losses?
• RQ5 How does the proposed aWAE compare to state-of-the-art models like Mult-VAE, Mult-DAE, CDAE, and Slim in terms of Recall and NDCG metrics?

Key findings

• The proposed aWAE achieves the best performance on Recall@1, Recall@5, and NDCG@10 across all three datasets (ML-20M, Netflix, LASTFM), outperforming Mult-VAE, Mult-DAE, CDAE, and Slim.
• On the LASTFM dataset, the aWAE achieves the best performance on five evaluation criteria and second-best on two, demonstrating strong generalization.
• The mutual information loss contributes stably across different hyperparameter settings, with optimal performance at α=0.3, indicating robustness to hyperparameter choice.
• The multinomial cost function enables faster convergence and significantly improves model performance by modeling non-clicked items, which are often ignored in standard autoencoders.
• The SMV method for approximating Wasserstein distance provides a stable and effective alternative to GAN-style discriminators, improving distributional alignment.
• The aWAE achieves comparable performance to Mult-VAE on Recall@20 and Recall@50 (e.g., 0.441 vs. 0.444), suggesting strong performance when recommending larger item sets.