[Paper Review] Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Bo Pang, Lillian Lee|ArXiv.org|Jun 17, 2005

Sentiment Analysis and Opinion Mining22 references726 citations

TL;DR

This paper proposes a metric labeling meta-algorithm that improves sentiment categorization on multi-point rating scales (e.g., 1–5 stars) by explicitly modeling label similarity—such as 'three stars' being closer to 'four stars' than 'one star'—and incorporating a novel item similarity measure based on positive sentence percentage. The method significantly outperforms standard SVM-based multi-class and regression approaches, especially when combined with this tailored similarity measure.

ABSTRACT

We address the rating-inference problem, wherein rather than simply decide whether a review is "thumbs up" or "thumbs down", as in previous sentiment analysis work, one must determine an author's evaluation with respect to a multi-point scale (e.g., one to five "stars"). This task represents an interesting twist on standard multi-class text categorization because there are several different degrees of similarity between class labels; for example, "three stars" is intuitively closer to "four stars" than to "one star". We first evaluate human performance at the task. Then, we apply a meta-algorithm, based on a metric labeling formulation of the problem, that alters a given n-ary classifier's output in an explicit attempt to ensure that similar items receive similar labels. We show that the meta-algorithm can provide significant improvements over both multi-class and regression versions of SVMs when we employ a novel similarity measure appropriate to the problem.

Motivation & Objective

• To address the rating-inference problem, where the goal is to predict a numerical rating (e.g., 1–5 stars) from text, rather than just binary sentiment.
• To investigate whether humans can reliably distinguish small differences in rating scores, validating the task’s feasibility.
• To improve multi-class sentiment classification by explicitly modeling the ordinal relationship between rating labels, such as proximity in scale.
• To develop and evaluate a meta-algorithm that modifies a base classifier’s output to ensure similar items receive similar labels, leveraging label and item similarity.
• To explore the effectiveness of different algorithmic approaches—one-vs-all, regression, and metric labeling—on this task.

Proposed method

• The paper formulates the rating-inference task as a metric labeling problem, where label similarity is encoded via a distance metric (e.g., |3−4| < |3−1|).
• A meta-algorithm is applied to reassign labels from a base classifier (e.g., SVM) to enforce that similar items receive similar labels, using graph-based optimization.
• A novel item similarity measure is proposed, based on the percentage of positive sentences in a review, to capture semantic similarity between reviews.
• The method is evaluated in both supervised and semi-supervised settings, with performance measured using accuracy on a five-star rating scale.
• The approach is compared against standard multi-class SVM, regression SVM, and one-vs-all SVM, with the metric labeling framework showing superior results when combined with the new similarity measure.
• The framework is extensible to other ordinal classification tasks, such as affect type, reading level, or urgency classification, by defining appropriate label metrics.

Experimental results

Research questions

• RQ1 Can human annotators reliably distinguish small differences in rating scores (e.g., 1-star vs. 2-star) when reviewing the same author’s work?
• RQ2 Does explicitly modeling label similarity (e.g., 'three stars' is closer to 'four stars' than to 'one star') improve performance in sentiment classification on rating scales?
• RQ3 Can a novel item similarity measure based on positive sentence percentage enhance the performance of a metric labeling-based classifier?
• RQ4 How does the metric labeling meta-algorithm compare to standard multi-class and regression approaches in terms of accuracy on the rating-inference task?
• RQ5 Can the metric labeling framework be generalized to other ordinal classification problems beyond sentiment rating?

Key findings

• Human subjects achieved 100% accuracy in distinguishing review pairs when the rating difference was three or more notches, validating the feasibility of fine-grained rating inference.
• The one-vs-all (OVA) SVM baseline achieved 65.4% accuracy on the 5-class rating task, while the metric labeling approach with the positive-sentence percentage (PSP) similarity measure improved it to 66.3%.
• Regression-based methods achieved 61.4% accuracy, which was outperformed by the OVA approach and further improved by the metric labeling framework with PSP to 61.5%.
• The metric labeling meta-algorithm significantly improved performance over standard multi-class and regression SVMs when combined with the novel PSP similarity measure.
• The method demonstrated robustness even when label calibration varied across authors, suggesting potential for generalization across different rating scales.
• The framework is extensible to other ordinal classification tasks, such as affect type, reading level, or urgency, by defining appropriate label distance metrics.