[Paper Review] Ensemble of Generative and Discriminative Techniques for Sentiment Analysis of Movie Reviews

Grégoire Mesnil, Tomáš Mikolov+2 more|arXiv (Cornell University)|Dec 17, 2014

Sentiment Analysis and Opinion MiningComputer Science7 references95 citations

TL;DR

This paper proposes an ensemble model combining generative language models, sentence embeddings, and reweighted bag-of-words features for sentiment analysis on movie reviews. By integrating a likelihood ratio from separate positive and negative n-gram language models with discriminative models like NB-SVM and sentence vectors, the method achieves a new state-of-the-art accuracy of 92.57% on the IMDB dataset, with full code publicly available for reproducibility and extension.

ABSTRACT

Sentiment analysis is a common task in natural language processing that aims to detect polarity of a text document (typically a consumer review). In the simplest settings, we discriminate only between positive and negative sentiment, turning the task into a standard binary classification problem. We compare several ma- chine learning approaches to this problem, and combine them to achieve the best possible results. We show how to use for this task the standard generative lan- guage models, which are slightly complementary to the state of the art techniques. We achieve strong results on a well-known dataset of IMDB movie reviews. Our results are easily reproducible, as we publish also the code needed to repeat the experiments. This should simplify further advance of the state of the art, as other researchers can combine their techniques with ours with little effort.

Motivation & Objective

• To improve sentiment classification performance on the IMDB movie review dataset using a diverse ensemble of machine learning techniques.
• To investigate whether generative language models, often overlooked in favor of discriminative models, can contribute meaningfully to sentiment analysis.
• To demonstrate that combining complementary models—especially including a generative approach—yields superior performance over individual models.
• To provide a fully reproducible framework by releasing code, enabling future researchers to easily integrate new models into the ensemble.

Proposed method

• Train two separate n-gram language models: one on positive reviews and one on negative reviews, using Kneser-Ney smoothing via the SRILM toolkit.
• Compute the likelihood ratio of a test review under the positive and negative language models as a discriminative feature, incorporating prior class probabilities via Bayes' rule.
• Use sentence embeddings (sentence vectors) trained via skip-gram with negative sampling to represent entire reviews as fixed-size dense vectors.
• Apply a reweighted bag-of-words approach (NB-SVM with tf-idf reweighting) to enhance traditional bag-of-words representations.
• Combine the predictions from the three models (generative likelihood ratio, sentence vectors, and NB-SVM) using a linear ensemble with learned weights.
• Optimize ensemble weights via grid search or alternative methods to maximize test accuracy, with all models trained and evaluated under the same protocol.

Experimental results

Research questions

• RQ1 Can generative language models, trained on positive and negative reviews separately, provide useful discriminative signals for sentiment classification?
• RQ2 How does the performance of a generative model compare to state-of-the-art discriminative models like NB-SVM and sentence embeddings on the IMDB sentiment dataset?
• RQ3 To what extent do diverse models—especially those with different inductive biases—complement each other in an ensemble setting?
• RQ4 Does combining multiple models, including a generative one, lead to a measurable improvement over the best individual model?

Key findings

• The ensemble of RNN language models, sentence vectors, and NB-SVM with trigram features achieved a new state-of-the-art accuracy of 92.57% on the IMDB dataset.
• The individual generative model (n-gram language model) performed the weakest, with an accuracy of 86.5%, but still contributed meaningfully to the ensemble.
• The NB-SVM trigram model alone achieved 91.87% accuracy, the highest among individual models, but was outperformed by the ensemble.
• The combination of sentence vectors and NB-SVM trigrams achieved 92.39% accuracy, showing strong synergy between these two models.
• Removal analysis showed that the generative model contributed the least individually, but its inclusion improved the ensemble, indicating complementary feature learning.
• The ensemble correctly classified reviews misclassified by individual models, such as subtle or sarcastic reviews, demonstrating robustness to nuanced sentiment.