[Paper Review] Predicting User Preferences

Pavel Sirotkin|arXiv (Cornell University)|Mar 15, 2011

Web Data Mining and Analysis12 references3 citations

TL;DR

This paper proposes a new measure to evaluate how well retrieval metrics predict actual user preferences in search results. Using this measure, it finds that Discounted Cumulative Gain (DCG) outperforms Mean Average Precision and classical precision, and counterintuitively, including more results can reduce a metric's predictive power.

ABSTRACT

The many metrics employed for the evaluation of search engine results have not themselves been conclusively evaluated. We propose a new measure for a metric's ability to identify user preference of result lists. Using this measure, we evaluate the metrics Discounted Cumulated Gain, Mean Average Precision and classical precision, finding that the former performs best. We also show that considering more results for a given query can impair rather than improve a metric's ability to predict user preferences.

Motivation & Objective

• To address the lack of conclusive evaluation for existing search evaluation metrics.
• To develop a new measure that quantitatively assesses how well a metric predicts actual user preferences.
• To compare the predictive performance of widely used metrics: DCG, MAP, and classical precision.
• To investigate the impact of result list length on a metric’s ability to predict user preferences.

Proposed method

• Propose a novel measure to evaluate a metric’s ability to predict user preferences based on user behavior data.
• Use real user preference data to calibrate and validate the proposed measure.
• Apply the measure to evaluate three standard metrics: Discounted Cumulative Gain (DCG), Mean Average Precision (MAP), and classical precision.
• Systematically vary the number of results considered per query to assess the effect on predictive performance.
• Use statistical analysis to compare the predictive accuracy of each metric across different list lengths.
• Validate findings using empirical data from a real-world search evaluation context.

Experimental results

Research questions

• RQ1 How well can standard retrieval metrics predict actual user preferences in search results?
• RQ2 Which metric—DCG, MAP, or classical precision—most accurately predicts user preferences?
• RQ3 Does increasing the number of results in a result list improve or impair a metric’s ability to predict user preferences?
• RQ4 What is the optimal list length for maximizing a metric’s predictive power?

Key findings

• Discounted Cumulative Gain (DCG) demonstrates the highest predictive performance among the evaluated metrics.
• Mean Average Precision (MAP) performs worse than DCG in predicting user preferences.
• Classical precision shows the weakest predictive ability among the three metrics tested.
• Increasing the number of results in a result list reduces the predictive power of all metrics, contrary to common intuition.
• The study reveals that longer result lists impair a metric’s ability to predict user preferences, suggesting diminishing returns beyond a certain list length.