[Paper Review] Towards Evaluating User Profiling Methods Based on Explicit Ratings on Item Features

Luca Luciano Costanzo, Yashar Deldjoo+3 more|arXiv (Cornell University)|Aug 29, 2019

Recommender Systems and Techniques23 references4 citations

TL;DR

This paper evaluates implicit user profiling methods against explicit user ratings on movie features (genres, actors, directors), revealing low similarity between inferred and actual user preferences. The authors create a publicly available dataset and find that even the best method achieves only 58.07% average cosine similarity for genres, indicating significant room for improvement in user profile modeling for recommender systems.

ABSTRACT

In order to improve the accuracy of recommendations, many recommender systems nowadays use side information beyond the user rating matrix, such as item content. These systems build user profiles as estimates of users' interest on content (e.g., movie genre, director or cast) and then evaluate the performance of the recommender system as a whole e.g., by their ability to recommend relevant and novel items to the target user. The user profile modelling stage, which is a key stage in content-driven RS is barely properly evaluated due to the lack of publicly available datasets that contain user preferences on content features of items. To raise awareness of this fact, we investigate differences between explicit user preferences and implicit user profiles. We create a dataset of explicit preferences towards content features of movies, which we release publicly. We then compare the collected explicit user feature preferences and implicit user profiles built via state-of-the-art user profiling models. Our results show a maximum average pairwise cosine similarity of 58.07\% between the explicit feature preferences and the implicit user profiles modelled by the best investigated profiling method and considering movies' genres only. For actors and directors, this maximum similarity is only 9.13\% and 17.24\%, respectively. This low similarity between explicit and implicit preference models encourages a more in-depth study to investigate and improve this important user profile modelling step, which will eventually translate into better recommendations.

Motivation & Objective

• To investigate the discrepancy between explicit user ratings on item features (e.g., genre, actors) and implicit user profiles derived from movie ratings.
• To evaluate the effectiveness of state-of-the-art user profiling methods in reconstructing true user preferences on item features.
• To address the lack of publicly available datasets containing explicit user preferences on item content features for evaluation purposes.
• To highlight the critical gap in evaluating the user profiling stage in content-based and hybrid recommender systems.
• To encourage deeper research into improving user profile modeling to enhance overall recommendation accuracy.

Proposed method

• Collected explicit user ratings on movie features (genres, actors, directors) via a crowdsourcing web application, creating a publicly available dataset.
• Constructed explicit user profiles as binary vectors where rated features have weight 1 and others 0, based on user responses.
• Applied four state-of-the-art user profiling methods—Zhang, Li, Symeonidis, and TF-IDF—on movie rating data to generate implicit user profiles.
• Computed pairwise similarity between explicit and implicit profiles using cosine and Jaccard similarity metrics.
• Evaluated similarity across three feature types: genres (19 categories), actors (567K), and directors (58K), with varying feature cardinalities.
• Used k-top feature selection for Jaccard similarity to handle scalar weights by focusing on the most relevant features per user.

Experimental results

Research questions

• RQ1 How similar are implicit user profiles generated by state-of-the-art methods to explicit user preferences on item features?
• RQ2 What is the impact of feature type (e.g., genre vs. actor) on the similarity between explicit and implicit user profiles?
• RQ3 How does the cardinality of feature sets (e.g., 19 genres vs. 567K actors) affect the performance of user profiling methods?
• RQ4 To what extent do existing user profiling methods accurately reconstruct true user tastes as expressed through explicit ratings?
• RQ5 Can the low similarity between explicit and implicit profiles be attributed to noise in user ratings or limitations in modeling techniques?

Key findings

• The best-performing user profiling method (Li) achieved only 58.07% average cosine similarity between implicit profiles and explicit user preferences for movie genres.
• For actors, the maximum average cosine similarity across all methods was only 9.13%, indicating poor alignment between implicit models and explicit tastes.
• For directors, the highest average cosine similarity was 17.24%, still significantly low, suggesting weak modeling of user preferences on this feature type.
• Jaccard similarity results were consistently lower than cosine similarity, with the best method achieving 36.19% for genres and only 5.73% for actors.
• The low similarity across all feature types underscores the limitations of current implicit user profiling methods in capturing true user preferences.
• The findings suggest that inaccurate user profile modeling may be a key bottleneck in recommendation accuracy, warranting further research into improved profiling techniques.