[Paper Review] A Comprehensive Survey of Evaluation Techniques for Recommendation Systems

Aryan Jadon, Avinash Patil|arXiv (Cornell University)|Dec 26, 2023

Recommender Systems and Techniques4 citations

TL;DR

This paper proposes a comprehensive, multi-dimensional framework for evaluating recommendation systems by categorizing evaluation metrics into five types: similarity, candidate generation, predictive, ranking, and business metrics. It demonstrates through empirical evaluation on MovieLens datasets that accuracy-focused metrics alone are insufficient, and that A/B testing and business-aligned metrics are essential for capturing real-world impact and user satisfaction.

ABSTRACT

The effectiveness of recommendation systems is pivotal to user engagement and satisfaction in online platforms. As these recommendation systems increasingly influence user choices, their evaluation transcends mere technical performance and becomes central to business success. This paper addresses the multifaceted nature of recommendations system evaluation by introducing a comprehensive suite of metrics, each tailored to capture a distinct aspect of system performance. We discuss * Similarity Metrics: to quantify the precision of content-based filtering mechanisms and assess the accuracy of collaborative filtering techniques. * Candidate Generation Metrics: to evaluate how effectively the system identifies a broad yet relevant range of items. * Predictive Metrics: to assess the accuracy of forecasted user preferences. * Ranking Metrics: to evaluate the effectiveness of the order in which recommendations are presented. * Business Metrics: to align the performance of the recommendation system with economic objectives. Our approach emphasizes the contextual application of these metrics and their interdependencies. In this paper, we identify the strengths and limitations of current evaluation practices and highlight the nuanced trade-offs that emerge when optimizing recommendation systems across different metrics. The paper concludes by proposing a framework for selecting and interpreting these metrics to not only improve system performance but also to advance business goals. This work is to aid researchers and practitioners in critically assessing recommendation systems and fosters the development of more nuanced, effective, and economically viable personalization strategies. Our code is available at GitHub - https://github.com/aryan-jadon/Evaluation-Metrics-for-Recommendation-Systems.

Motivation & Objective

• To address the lack of a unified, holistic evaluation framework for recommendation systems that considers multiple performance dimensions.
• To identify the limitations of relying solely on accuracy-based metrics and highlight the importance of diversity, novelty, and business impact.
• To bridge the gap between machine learning model performance and real-world business outcomes through integrated evaluation.
• To provide practitioners and researchers with a structured, context-aware approach to selecting and interpreting evaluation metrics.
• To advocate for the integration of A/B testing and user feedback in evaluation to ensure alignment with business goals and user satisfaction.

Proposed method

• Proposes a five-tiered evaluation framework: similarity, candidate generation, predictive, ranking, and business metrics, each targeting distinct aspects of system performance.
• Classifies and defines key metrics within each category, including cosine similarity, Jaccard index, RMSE, MAE, nDCG@k, MAP, and MRR.
• Employs empirical evaluation on two MovieLens datasets (100k and 1M) to compare performance across seven algorithms: ALS, SAR, SVD, NCF, BPR, BiVAE, and LightGCN.
• Uses standard evaluation protocols with k=10 for ranking metrics and reports training time, inference time, and multiple error and ranking metrics.
• Integrates A/B testing as a critical validation method to connect model predictions with real-world business KPIs such as click-through rate and revenue.
• Emphasizes contextual metric selection, highlighting trade-offs between metrics (e.g., accuracy vs. diversity) and advocating for a balanced, goal-driven evaluation strategy.

Experimental results

Research questions

• RQ1 How can recommendation system evaluation be systematically categorized to cover all critical performance dimensions beyond accuracy?
• RQ2 What are the strengths and limitations of current evaluation practices, particularly in terms of trade-offs between different metrics?
• RQ3 To what extent do traditional predictive metrics (e.g., RMSE, MAE) correlate with real-world business outcomes and user satisfaction?
• RQ4 How do ranking metrics like nDCG@k and MAP reflect the quality of recommendation lists in practical deployment?
• RQ5 What role does A/B testing play in validating the business impact of recommendation systems beyond offline model evaluation?

Key findings

• Ranking metrics such as nDCG@k and MAP provide more meaningful insights into recommendation quality than accuracy-only metrics like RMSE or MAE.
• LightGCN achieved the highest nDCG@k (0.4216) and MAP (0.1227) on the MovieLens 100k dataset, outperforming ALS, SVD, and BPR.
• On the MovieLens 1M dataset, BiVAE and LightGCN achieved the highest nDCG@k (0.4247 and 0.4241 respectively), indicating strong performance in ranking quality.
• Business metrics such as click-through rate and revenue showed significant improvements when A/B testing was used, confirming that model performance does not always translate directly to business value.
• The study reveals that optimizing for predictive accuracy (e.g., low RMSE) may compromise diversity and novelty, which are critical for long-term user engagement.
• A/B testing is essential for validating the real-world impact of recommendation systems, as it connects model predictions with tangible business outcomes and user feedback.