[Paper Review] Training Deep AutoEncoders for Collaborative Filtering

Oleksii Kuchaiev, Boris Ginsburg|arXiv (Cornell University)|Aug 5, 2017

Music and Audio Processing13 references60 citations

TL;DR

The paper trains very deep autoencoders for collaborative filtering, introducing dense re-feeding and high dropout with SELU activations to achieve state-of-the-art future rating prediction on Netflix data without pre-training.

ABSTRACT

This paper proposes a novel model for the rating prediction task in recommender systems which significantly outperforms previous state-of-the art models on a time-split Netflix data set. Our model is based on deep autoencoder with 6 layers and is trained end-to-end without any layer-wise pre-training. We empirically demonstrate that: a) deep autoencoder models generalize much better than the shallow ones, b) non-linear activation functions with negative parts are crucial for training deep models, and c) heavy use of regularization techniques such as dropout is necessary to prevent over-fiting. We also propose a new training algorithm based on iterative output re-feeding to overcome natural sparseness of collaborate filtering. The new algorithm significantly speeds up training and improves model performance. Our code is available at https://github.com/NVIDIA/DeepRecommender

Motivation & Objective

• Motivate improved rating prediction in recommender systems using deep autoencoders.
• Demonstrate that deeper architectures generalize better than shallow ones for CF.
• Show the importance of non-linear activations with negative parts and strong regularization.
• Introduce iterative dense re-feeding to address data sparsity and speed training.

Proposed method

• Use a deep autoencoder with encoder and decoder mirrors to learn dense representations from sparse user rating vectors.
• Train end-to-end without layer-wise pre-training.
• Employ SELU activations, high dropout, and constrained (tied) weights to reduce overfitting.
• Introduce iterative output re-feeding to create dense training signals from sparse inputs.
• Optimize masked mean squared error (MMSE) to ignore zeros in input vectors.
• Experiment with different depths, dropouts, and training regimens, including a higher learning rate combined with re-feeding for improved performance.

Experimental results

Research questions

• RQ1 Can deeper autoencoder architectures improve future rating prediction over shallower models in collaborative filtering?
• RQ2 How do activation functions, particularly those with negative parts, affect training of deep CF autoencoders?
• RQ3 What regularization and training strategies (e.g., dropout, tied weights, dense re-feeding) yield better generalization and faster convergence?
• RQ4 Does iterative dense re-feeding allow training with higher learning rates while maintaining or improving performance?

Key findings

• Deeper autoencoder architectures improve evaluation RMSE compared to shallower ones on Netflix data.
• Activation functions with non-zero negative parts and unbounded positive parts (e.g., SELU) outperform Sigmoid, ReLU, and Tanh on this task.
• High dropout (e.g., 0.8) in the coding layer helps prevent overfitting and improves generalization.
• Iterative dense re-feeding combined with a higher learning rate significantly improves performance over baseline training.
• The best DeepRec model on Netflix Full achieves a test RMSE of 0.9099 in their experiments, outperforming prior methods like I-AR, U-AR, and RRN in their reported setup.
• Deeper models with appropriate regularization can surpass temporal dynamics methods for future rating prediction on the evaluated data.