[Paper Review] A Neural Autoregressive Approach to Collaborative Filtering

Yin Zheng, Bangsheng Tang+2 more|arXiv (Cornell University)|May 31, 2016

Music and Audio ProcessingComputer Science22 references150 citations

TL;DR

CF-NADE is a neural autoregressive model for collaborative filtering that shares parameters across ratings, uses a factored version for scalability, includes an ordinal cost, and can be extended to deep architectures; it achieves state-of-the-art RMSE on MovieLens and Netflix benchmarks.

ABSTRACT

This paper proposes CF-NADE, a neural autoregressive architecture for collaborative filtering (CF) tasks, which is inspired by the Restricted Boltzmann Machine (RBM) based CF model and the Neural Autoregressive Distribution Estimator (NADE). We first describe the basic CF-NADE model for CF tasks. Then we propose to improve the model by sharing parameters between different ratings. A factored version of CF-NADE is also proposed for better scalability. Furthermore, we take the ordinal nature of the preferences into consideration and propose an ordinal cost to optimize CF-NADE, which shows superior performance. Finally, CF-NADE can be extended to a deep model, with only moderately increased computational complexity. Experimental results show that CF-NADE with a single hidden layer beats all previous state-of-the-art methods on MovieLens 1M, MovieLens 10M, and Netflix datasets, and adding more hidden layers can further improve the performance.

Motivation & Objective

• Introduce CF-NADE, a neural autoregressive model for collaborative filtering inspired by NADE and RBM-CF.
• Improve performance by sharing parameters across ratings and by using a factored parameterization for scalability.
• Incorporate the ordinal nature of ratings via an ordinal cost to improve prediction quality.
• Extend CF-NADE to a deep architecture to enhance performance with manageable computation.
• Evaluate CF-NADE on MovieLens 1M, MovieLens 10M, and Netflix to demonstrate state-of-the-art results.

Proposed method

• Model user-specific CF-NADEs that share parameters across users with a fixed architecture.
• Compute conditional rating probabilities p(r_moi|r_{<m_i}) via neural representations and scoring functions (Equations 3 and 4).
• Share parameters across ratings to form cumulative usage of W^k, V^k, and b^k (Equations 9 and 10).
• Factorize large weight matrices W^k and V^k into low-rank products to reduce parameters (Equations 11 and 12).
• Introduce an ordinal cost C_ord and a hybrid objective C_hybrid to exploit rating order (Equation 18).
• Extend to deep CF-NADE by stochastic ordering and hierarchical hidden layers with a training objective that aligns with deep NADE adaptations (Equations 19-21).

Experimental results

Research questions

• RQ1 Can a neural autoregressive model (CF-NADE) effectively model user-item rating vectors while sharing parameters across ratings?
• RQ2 Does parameter sharing and a factored representation improve scalability and performance on large CF benchmarks?
• RQ3 Does incorporating ordinal information via an ordinal cost improve predictive accuracy for ratings?
• RQ4 Can CF-NADE be extended to a deep architecture without prohibitive computational costs while maintaining performance?
• RQ5 How does CF-NADE perform against state-of-the-art CF methods on MovieLens 1M, MovieLens 10M, and Netflix datasets?

Key findings

• CF-NADE with shared ratings (CF-NADE-S) achieves competitive to superior RMSE compared to baselines on MovieLens 1M.
• Adding a second hidden layer (deep CF-NADE) yields further RMSE improvement over shallow variants.
• On MovieLens 1M, I-CF-NADE-S (single layer) achieves RMSE 0.830 and I-CF-NADE-S (2 layers) 0.829, outperforming several baselines.
• On MovieLens 10M, U-CF-NADE-S (single layer) achieves RMSE 0.772 and with two layers 0.771, beating multiple baselines.
• On Netflix, U-CF-NADE-S (single layer) achieves RMSE 0.804 and 0.803 with two layers, outperforming all listed baselines.
• A factored version reduces parameter count dramatically with little or no loss in performance on large datasets.