[Paper Review] Thumbs up? Sentiment Classification using Machine Learning Techniques

Bo Pang, Lillian Lee+1 more|ArXiv.org|May 28, 2002

Sentiment Analysis and Opinion MiningComputer Science21 references2,209 citations

TL;DR

The paper evaluates three standard supervised learning methods (Naive Bayes, Maximum Entropy, and SVM) for sentiment classification on movie reviews and analyzes factors influencing performance, showing these methods surpass baselines but underperform topic-based categorization.

ABSTRACT

We consider the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative. Using movie reviews as data, we find that standard machine learning techniques definitively outperform human-produced baselines. However, the three machine learning methods we employed (Naive Bayes, maximum entropy classification, and support vector machines) do not perform as well on sentiment classification as on traditional topic-based categorization. We conclude by examining factors that make the sentiment classification problem more challenging.

Motivation & Objective

• Evaluate the effectiveness of standard machine learning methods for sentiment classification versus human baselines.
• Assess how feature choices (unigrams, bigrams, presence vs frequency, negation handling) affect performance.
• Understand challenges that make sentiment classification harder than topic-based categorization.
• Provide corpus-based baselines and insights into linguistic patterns affecting sentiment analysis.

Proposed method

• Represent each document as a bag-of-features vector with predefined features such as unigrams and bigrams.
• Train Naive Bayes, Maximum Entropy, and SVM classifiers on labeled movie reviews with three-fold cross-validation.
• Investigate the impact of feature frequency versus presence information on performance.
• Incorporate negation tagging to capture contextual sentiment shifts.
• Evaluate the effect of additional features (POS tags, adjectives, position) on accuracy.

Experimental results

Research questions

• RQ1 Can standard supervised learning methods achieve high accuracy in sentiment classification on movie reviews?
• RQ2 How do feature choices (unigrams vs bigrams, presence vs frequency) impact classification performance?
• RQ3 Does incorporating negation tagging or POS information improve sentiment detection?
• RQ4 Is sentiment classification as easy as topic categorization, or are additional techniques required to handle discourse and contextual cues?

Key findings

• All three learners (NB, MaxEnt, SVM) outperform random baselines and human unigram baselines on sentiment data.
• Using feature presence (binary) generally yields higher accuracy than using feature frequency for NB and SVM.
• Bigrams do not improve performance beyond unigram presence and can reduce accuracy when used alone.
• Negation tagging offers slight gains for NB but not consistently across models.
• Adjectives alone provide weaker information than unigrams; top unigrams perform comparably to broader feature sets.
• Overall, sentiment classification remains more challenging than topic-based categorization, with best results still lagging behind high-accuracy topic models.