[Paper Review] Nonlinear Inductive Matrix Completion based on One-layer Neural Networks

Kai Zhong, Zhao Song+2 more|arXiv (Cornell University)|May 26, 2018

Recommender Systems and Techniques17 references3 citations

TL;DR

This paper proposes a nonlinear inductive matrix completion framework using one-layer neural networks with non-linear activation functions (e.g., sigmoid, tanh) to improve recommendation accuracy beyond linear models. It establishes that gradient descent converges to the global optimum when initialized near the true parameters, leveraging strong convexity in the neighborhood, and demonstrates superior performance on recommendation and clustering tasks compared to linear IMC.

ABSTRACT

The goal of a recommendation system is to predict the interest of a user in a given item by exploiting the existing set of ratings as well as certain user/item features. A standard approach to modeling this problem is Inductive Matrix Completion where the predicted rating is modeled as an inner product of the user and the item features projected onto a latent space. In order to learn the parameters effectively from a small number of observed ratings, the latent space is constrained to be low-dimensional which implies that the parameter matrix is constrained to be low-rank. However, such bilinear modeling of the ratings can be limiting in practice and non-linear prediction functions can lead to significant improvements. A natural approach to introducing non-linearity in the prediction function is to apply a non-linear activation function on top of the projected user/item features. Imposition of non-linearities further complicates an already challenging problem that has two sources of non-convexity: a) low-rank structure of the parameter matrix, and b) non-linear activation function. We show that one can still solve the non-linear Inductive Matrix Completion problem using gradient descent type methods as long as the solution is initialized well. That is, close to the optima, the optimization function is strongly convex and hence admits standard optimization techniques, at least for certain activation functions, such as Sigmoid and tanh. We also highlight the importance of the activation function and show how ReLU can behave significantly differently than say a sigmoid function. Finally, we apply our proposed technique to recommendation systems and semi-supervised clustering, and show that our method can lead to much better performance than standard linear Inductive Matrix Completion methods.

Motivation & Objective

• To address the limitations of linear inductive matrix completion (IMC) in modeling complex user-item relationships.
• To develop a nonlinear extension of IMC using one-layer neural networks with non-linear activations to better capture non-linear patterns in user and item features.
• To provide theoretical guarantees for optimization convergence in the non-convex, non-linear setting by showing strong convexity near optimal solutions.
• To empirically validate the method's superiority over linear IMC in recommendation and semi-supervised clustering tasks.

Proposed method

• Formulates nonlinear inductive matrix completion (NIMC) as a prediction function using non-linear mappings: A(x,y) = ⟨ϕ(Uᵀx), ϕ(Vᵀy)⟩, where ϕ is a non-linear activation function.
• Uses gradient descent for parameter optimization, with theoretical analysis showing convergence when initialization is sufficiently close to the true parameters.
• Applies matrix concentration inequalities and Lipschitz continuity of activation functions (e.g., sigmoid, tanh) to bound estimation errors in the Hessian and gradient terms.
• Analyzes the Hessian of the loss function and proves strong convexity in a neighborhood of the optimal solution for certain activation functions.
• Compares ReLU with sigmoid/tanh, highlighting differences in behavior and convergence properties due to non-negativity constraints in ReLU.
• Employs a data-dependent sampling model with observed user-item rating pairs Ω to train the model efficiently with limited data.

Experimental results

Research questions

• RQ1 Can a one-layer neural network-based inductive matrix completion model achieve better recommendation accuracy than linear IMC by introducing non-linearity?
• RQ2 Under what conditions does gradient descent converge to the global optimum in this non-convex, non-linear setting?
• RQ3 How do different activation functions (e.g., ReLU vs. sigmoid/tanh) affect optimization behavior and model performance?
• RQ4 What theoretical guarantees can be provided for the convergence of the optimization process in this nonlinear framework?
• RQ5 How does the model perform on real-world recommendation and semi-supervised clustering tasks compared to linear IMC?

Key findings

• The proposed NIMC model with sigmoid or tanh activations achieves significantly better performance on recommendation and clustering tasks than standard linear IMC.
• Gradient descent converges to the global optimum when initialized near the true parameters, due to strong convexity of the loss function in that region.
• Theoretical analysis shows that the Hessian of the loss function is well-conditioned near the optimum, enabling efficient optimization.
• ReLU-based models behave differently due to the non-negativity constraint of the latent space, which can be beneficial or limiting depending on the data.
• The method requires only a small number of observed ratings and maintains low parameter count, making it suitable for data-scarce settings.
• Empirical results confirm that the nonlinear model outperforms linear IMC across multiple benchmark datasets in both recommendation and clustering tasks.