[论文解读] Learning to Generate Reviews and Discovering Sentiment

Alec Radford, Rafał Józefowicz+1 more|arXiv (Cornell University)|Apr 5, 2017

Topic Modeling参考文献 47被引用 350

一句话总结

论文表明，按字节级别的乘性LSTM学习到一个可解释的情感单元，能够预测情感并生成情感控制的文本，在若干情感任务上通过无监督预训练获得强劲结果。

ABSTRACT

We explore the properties of byte-level recurrent language models. When given sufficient amounts of capacity, training data, and compute time, the representations learned by these models include disentangled features corresponding to high-level concepts. Specifically, we find a single unit which performs sentiment analysis. These representations, learned in an unsupervised manner, achieve state of the art on the binary subset of the Stanford Sentiment Treebank. They are also very data efficient. When using only a handful of labeled examples, our approach matches the performance of strong baselines trained on full datasets. We also demonstrate the sentiment unit has a direct influence on the generative process of the model. Simply fixing its value to be positive or negative generates samples with the corresponding positive or negative sentiment.

研究动机与目标

• Investigate whether unsupervised byte-level language models can learn meaningful high-level concepts like sentiment.
• Assess data efficiency and the quality of representations learned for sentiment-related tasks.
• Examine the existence and utility of a disentangled sentiment unit within a large-scale language model.
• Explore how sentiment information influences the generative process of the model.
• Evaluate cross-domain and dataset limitations to understand the boundaries of unsupervised representations.

提出的方法

• Train a single-layer multiplicative LSTM (mLSTM) with 4096 units on a large Amazon product review corpus (~82 million reviews).
• Process text as UTF-8 bytes and use the final cell state as a fixed feature representation for downstream tasks.
• Train a logistic regression classifier on top of the mLSTM representation for sentiment and related tasks.
• Apply L1 regularization to improve performance in low-data regimes and identify sparse, interpretable features.
• Analyze and visualize the sentiment-related unit learned within the mLSTM and its impact on generation by fixing its value.

实验结果

研究问题

• RQ1 Can byte-level language models learn disentangled high-level concepts such as sentiment without supervision?
• RQ2 How data-efficient can such representations be for sentiment analysis compared to supervised baselines?
• RQ3 Is there a single unit that captures sentiment, and can it meaningfully influence text generation?
• RQ4 What are the limitations of such unsupervised representations when transferred to tasks beyond sentiment in-domain?
• RQ5 How does domain and dataset distribution affect the learned sentiment representation and model performance?

主要发现

• A single sentiment-disentangled unit emerges within the mLSTM, with a bimodal activation distribution that separates positive and negative sentiment.
• The sentiment unit alone achieves 92.30% test accuracy on IMDB when thresholded, outperforming NB-SVM trigram and approaching semi-supervised state-of-the-art.
• The full 4096-unit representation yields 92.88% accuracy on IMDB, offering only a small gain over the single sentiment unit.
• On binary SST, the unsupervised representation matches state-of-the-art results with a fraction of labeled data and is data-efficient (visible in Figure 2).
• The model exhibits a capacity ceiling on large, out-of-domain datasets (Yelp), achieving 95.22% on Yelp with full data but remaining competitive with simpler baselines in some settings.
• Fixing the sentiment unit to positive or negative can steer generation toward corresponding sentiment in sampled reviews (demonstrating controllable text generation).
• The learned representation is most effective in domain-similar sentiment tasks (MR, CR) and less so for generic semantic relatedness or out-of-domain tasks (SICK).