[Paper Review] Fifty Shades of Ratings: How to Benefit from a Negative Feedback in Top-N Recommendations Tasks

Evgeny Frolov, Ivan Oseledets|arXiv (Cornell University)|Jul 14, 2016

Tensor decomposition and applications20 references3 citations

TL;DR

This paper proposes a tensor-based collaborative filtering model, CoFFee, that treats user ratings as a ternary categorical variable (positive, neutral, negative) rather than a cardinal utility, enabling effective learning from purely negative feedback. By using third-order tensor factorization and a higher-order folding-in method, the model achieves state-of-the-art performance in top-N recommendations, especially in cold-start scenarios with no positive feedback, and outperforms standard metrics by revealing biases in recommendation quality.

ABSTRACT

Conventional collaborative filtering techniques treat a top-n recommendations problem as a task of generating a list of the most relevant items. This formulation, however, disregards an opposite - avoiding recommendations with completely irrelevant items. Due to that bias, standard algorithms, as well as commonly used evaluation metrics, become insensitive to negative feedback. In order to resolve this problem we propose to treat user feedback as a categorical variable and model it with users and items in a ternary way. We employ a third-order tensor factorization technique and implement a higher order folding-in method to support online recommendations. The method is equally sensitive to entire spectrum of user ratings and is able to accurately predict relevant items even from a negative only feedback. Our method may partially eliminate the need for complicated rating elicitation process as it provides means for personalized recommendations from the very beginning of an interaction with a recommender system. We also propose a modification of standard metrics which helps to reveal unwanted biases and account for sensitivity to a negative feedback. Our model achieves state-of-the-art quality in standard recommendation tasks while significantly outperforming other methods in the cold-start "no-positive-feedback" scenarios.

Motivation & Objective

• Address the limitation of conventional collaborative filtering in handling negative feedback, which often leads to irrelevant recommendations and poor user engagement.
• Overcome the insensitivity of standard evaluation metrics (e.g., nDCG) to irrelevant recommendations, which can mask poor performance in cold-start or negative-feedback scenarios.
• Develop a unified recommendation model that performs well from the very beginning of user interaction, even without positive feedback.
• Introduce a modified evaluation metric (nDCL) that captures sensitivity to negative feedback and reveals biases in recommendation quality.

Proposed method

• Model user feedback as a categorical variable with three levels: positive, neutral, and negative, rather than treating ratings as a cardinal or ordinal utility.
• Employ third-order tensor factorization to jointly model users, items, and feedback categories, enabling the capture of complex user-item preferences across all feedback types.
• Implement a higher-order folding-in technique to support efficient online inference and personalized recommendations without retraining.
• Introduce a modified evaluation metric, nDCL (normalized Discounted Cumulative Loss), which penalizes the ranking of irrelevant items and reveals sensitivity to negative feedback.
• Calibrate the model using a rating prediction task to validate its ability to predict both exact ratings and rating positivity.
• Use a unified framework that supports both cold-start and standard recommendation scenarios, eliminating the need for separate rating elicitation procedures.

Experimental results

Research questions

• RQ1 Can a recommender system achieve high-quality recommendations using only negative feedback, without requiring positive ratings?
• RQ2 How do standard evaluation metrics like nDCG fail to detect poor performance when irrelevant items are ranked highly, especially in cold-start settings?
• RQ3 To what extent can a tensor-based model that treats feedback as a categorical variable outperform traditional matrix factorization in both positive and negative feedback scenarios?
• RQ4 Can a modified evaluation metric like nDCL effectively reveal biases in recommendation systems related to negative feedback?

Key findings

• The CoFFee model achieves state-of-the-art performance in standard top-N recommendation tasks, outperforming SVD and WRMF in most metrics, including nDCG and nDCL.
• In cold-start scenarios with no positive feedback, CoFFee significantly outperforms other models, demonstrating its ability to generate personalized recommendations from the very first interaction.
• The model correctly predicts the exact rating value in 47% of cases and the rating positivity in 48% of cases on the Movielens 1M dataset, achieving a 0.77 RMSE score.
• Standard metrics like nDCG fail to detect poor performance when irrelevant items are ranked highly; CoFFee’s nDCL metric reveals that SVD and WRMF models rank irrelevant items more highly than CoFFee.
• The model successfully recommends 'opposite' features (e.g., comedy instead of crime) based on negative feedback, as shown in examples like recommending Toy Story over Scarface.
• The model’s performance is robust even when users provide only negative feedback, proving that negative feedback alone can be sufficient for high-quality, personalized recommendations.