[Paper Review] Thumbs Up or Thumbs Down? Semantic Orientation Applied to Unsupervised Classification of Reviews

Peter D. Turney|ArXiv.org|Dec 11, 2002

Sentiment Analysis and Opinion Mining9 references1,580 citations

TL;DR

This paper introduces an unsupervised algorithm that classifies product reviews as positive or negative based on the semantic orientation of adjectives and adverbs in review phrases. It computes semantic orientation using mutual information between phrases and the words 'excellent' and 'poor', achieving 74% average accuracy on 410 reviews across four domains, with performance ranging from 66% for movies to 84% for automobiles.

ABSTRACT

This paper presents a simple unsupervised learning algorithm for classifying reviews as recommended (thumbs up) or not recommended (thumbs down). The classification of a review is predicted by the average semantic orientation of the phrases in the review that contain adjectives or adverbs. A phrase has a positive semantic orientation when it has good associations (e.g., "subtle nuances") and a negative semantic orientation when it has bad associations (e.g., "very cavalier"). In this paper, the semantic orientation of a phrase is calculated as the mutual information between the given phrase and the word "excellent" minus the mutual information between the given phrase and the word "poor". A review is classified as recommended if the average semantic orientation of its phrases is positive. The algorithm achieves an average accuracy of 74% when evaluated on 410 reviews from Epinions, sampled from four different domains (reviews of automobiles, banks, movies, and travel destinations). The accuracy ranges from 84% for automobile reviews to 66% for movie reviews.

Motivation & Objective

• To develop an unsupervised method for classifying reviews as recommended (thumbs up) or not recommended (thumbs down) without labeled training data.
• To identify semantic orientation of phrases using word associations with 'excellent' and 'poor' as polarity anchors.
• To evaluate the effectiveness of semantic orientation in predicting review sentiment across diverse domains such as automobiles, banks, movies, and travel.
• To demonstrate that mutual information between phrases and polarity anchor words can serve as a reliable unsupervised sentiment measure.
• To achieve high classification accuracy using only distributional semantic statistics, avoiding manual annotation.

Proposed method

• Semantic orientation of a phrase is calculated as the difference between mutual information with 'excellent' and with 'poor'.
• The algorithm extracts adjectives and adverbs from review phrases and computes their semantic orientation using co-occurrence statistics in a large corpus.
• A review is classified as positive if the average semantic orientation of its phrases is positive.
• Mutual information is used as a measure of association between a phrase and the polarity anchor words 'excellent' and 'poor'.
• The method relies solely on distributional statistics from a large text corpus, requiring no labeled sentiment data.
• The approach is applied to 410 reviews from Epinions, spanning four domains: automobiles, banks, movies, and travel destinations.

Experimental results

Research questions

• RQ1 Can semantic orientation derived from co-occurrence statistics with 'excellent' and 'poor' effectively classify review sentiment without labeled data?
• RQ2 How accurate is the unsupervised semantic orientation method in distinguishing positive from negative reviews across diverse domains?
• RQ3 Does the performance of the method vary significantly across different review domains such as movies versus automobiles?
• RQ4 To what extent can mutual information between phrases and polarity anchor words serve as a proxy for sentiment?
• RQ5 Can a simple, unsupervised algorithm achieve high accuracy in sentiment classification using only distributional semantics?

Key findings

• The algorithm achieved an average accuracy of 74% across 410 reviews from Epinions, demonstrating strong performance without any labeled training data.
• The highest accuracy was 84% for automobile reviews, indicating strong domain-specific performance in product categories with clear evaluative language.
• The lowest accuracy was 66% for movie reviews, suggesting challenges in sentiment detection for more subjective or nuanced domains.
• The method successfully leveraged distributional semantics through mutual information to infer sentiment polarity using only 'excellent' and 'poor' as anchors.
• The results confirm that semantic orientation based on co-occurrence with polarity words can serve as an effective unsupervised sentiment classifier.
• The approach demonstrates that unsupervised sentiment classification is feasible using only statistical associations with minimal lexical resources.