[Paper Review] Toward Controlled Generation of Text

Zhiting Hu, Zichao Yang+3 more|arXiv (Cornell University)|Mar 2, 2017

Topic Modeling34 references149 citations

TL;DR

The paper presents a VAE-based text generator augmented with holistic attribute discriminators to achieve disentangled, controllable sentence generation, using a wake-sleep style training and differentiable softmax approximations for discrete text.

ABSTRACT

Generic generation and manipulation of text is challenging and has limited success compared to recent deep generative modeling in visual domain. This paper aims at generating plausible natural language sentences, whose attributes are dynamically controlled by learning disentangled latent representations with designated semantics. We propose a new neural generative model which combines variational auto-encoders and holistic attribute discriminators for effective imposition of semantic structures. With differentiable approximation to discrete text samples, explicit constraints on independent attribute controls, and efficient collaborative learning of generator and discriminators, our model learns highly interpretable representations from even only word annotations, and produces realistic sentences with desired attributes. Quantitative evaluation validates the accuracy of sentence and attribute generation.

Motivation & Objective

• Motivate the need for controllable, realistic text generation with interpretable latent representations.
• Develop a neural model that enforces disentangled, independent attribute controls in generated sentences.
• Enable semi-supervised learning by leveraging discriminators and wake-sleep collaboration.
• Demonstrate controlled generation of sentiment and tense and show plausible short sentences.
• Show that word-level labels can induce sentence-level attribute control and disentanglement.

Proposed method

• Extend VAEs with structured latent codes c for explicit attributes (e.g., sentiment, tense).
• Attach a discriminator per attribute to guide generation via a differentiable softmax approximation for discrete text.
• Use the encoder as an additional discriminator to encourage independence of z (unstructured code) and c.
• Adopt an extended wake-sleep training loop where generated samples help train both generator and discriminators.
• Allow semi-supervised learning by training discriminators on labeled data while using unlabeled data for VAE reconstruction.
• Incorporate a minimum-entropy regularization to improve robustness of discriminator predictions on generated samples.

Experimental results

Research questions

• RQ1 Can disentangled, attribute-specific latent codes control generated text without entangling with other features?
• RQ2 Does a holistic attribute discriminator framework improve controllable text generation compared to reconstruction-based methods?
• RQ3 How effectively can semi-supervised learning leverage limited attribute labels to learn controllable generation?
• RQ4 Is an independency constraint between structured and unstructured latent codes achievable and beneficial for interpretability?
• RQ5 Can word-level or partial annotations suffice to induce sentence-level attribute control (e.g., sentiment, tense)?

Key findings

• The proposed model improves sentiment-attribute generation accuracy compared with S-VAE across SST-full, SST-small, and Lexicon datasets (0.851 vs 0.822 on SST-full; 0.707 vs 0.679 on SST-small; 0.701 vs 0.660 on Lexicon).
• Disentangled representations with an independency constraint yield more interpretable changes when varying attribute codes compared to models without the constraint.
• The model learns useful sentiment and tense control with only word-level labels or few labeled sentences, enabling effective sentence-level attribute manipulation.
• Generated samples can be used to augment training data and improve downstream sentiment classifiers.
• The sleep/wake-like training procedure enables efficient mutual bootstrapping between generator and discriminators, achieving semi-supervised learning with limited supervision.