[Paper Review] Generative and Discriminative Text Classification with Recurrent Neural Networks

Dani Yogatama, Chris Dyer+2 more|arXiv (Cornell University)|Mar 6, 2017

Topic ModelingComputer Science110 citations

TL;DR

The paper compares generative and discriminative LSTM-based text classifiers, showing discriminative models have lower asymptotic error, but generative models learn faster with less data and are more robust to distribution shifts in continual and zero-shot learning settings.

ABSTRACT

We empirically characterize the performance of discriminative and generative LSTM models for text classification. We find that although RNN-based generative models are more powerful than their bag-of-words ancestors (e.g., they account for conditional dependencies across words in a document), they have higher asymptotic error rates than discriminatively trained RNN models. However we also find that generative models approach their asymptotic error rate more rapidly than their discriminative counterparts---the same pattern that Ng & Jordan (2001) proved holds for linear classification models that make more naive conditional independence assumptions. Building on this finding, we hypothesize that RNN-based generative classification models will be more robust to shifts in the data distribution. This hypothesis is confirmed in a series of experiments in zero-shot and continual learning settings that show that generative models substantially outperform discriminative models.

Motivation & Objective

• Assess performance of discriminative vs. generative LSTM models on text classification tasks.
• Determine asymptotic error and sample complexity differences between model types.
• Evaluate robustness to distribution shifts via continual and zero-shot learning experiments.

Proposed method

• Implement discriminative LSTM with peephole connections and average hidden state pooling for document representation.
• Develop generative models as class-based language models with shared and independent LSTMs to compute p(x|y) and p(y).
• Train discriminative models to maximize p(y|x) and generative models to maximize p(x|y)p(y), using Bayes rule at prediction.
• Share word embeddings and LSTM parameters across classes in the Shared LSTM variant of the generative model.
• Compare with baselines including Naive Bayes, Kneser–Ney, and NB neural network across six datasets.
• Pretrain shared components on unlabeled data and fine-tune class embeddings for the Generative Shared LSTM.

Experimental results

Research questions

• RQ1 Do discriminative LSTMs achieve lower asymptotic error than generative LSTMs for text classification?
• RQ2 Do generative LSTMs exhibit faster learning and better generalization in small-data regimes?
• RQ3 Are generative models more robust to data distribution shifts in continual and zero-shot learning settings?

Key findings

• Discriminative LSTM achieves higher asymptotic accuracy than generative LSTM, aligning with Ng & Jordan (2001) results for linear models.
• Generative LSTMs reach their asymptotic error more quickly and outperform baseline generative models with stronger independence assumptions in small-data scenarios.
• In continual learning, generative models better handle sequentially introduced classes and mitigate catastrophic forgetting.
• In zero-shot learning, generative LSTM with fixed label embeddings demonstrates notable precision and recall on unseen classes, outperforming discriminative models in several settings when using self-training.
• Generative shared LSTM, pretrained on unlabeled data, approaches the performance of independently trained generative models while offering faster adaptation to new classes.