[Paper Review] Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space

Chunyuan Li, Xiang Gao+5 more|arXiv (Cornell University)|Apr 5, 2020

Topic Modeling56 references11 citations

TL;DR

Optimus proposes a large-scale, deep variational autoencoder (VAE) for natural language that pre-trains a universal latent space on massive text data, enabling both controllable text generation and robust zero-shot transfer to low-resource tasks. By combining the strengths of BERT and GPT-2, Optimus achieves state-of-the-art results in VAE language modeling, lower perplexity than GPT-2, and effective guided generation via latent vector arithmetic.

ABSTRACT

When trained effectively, the Variational Autoencoder (VAE) can be both a powerful generative model and an effective representation learning framework for natural language. In this paper, we propose the first large-scale language VAE model, Optimus. A universal latent embedding space for sentences is first pre-trained on large text corpus, and then fine-tuned for various language generation and understanding tasks. Compared with GPT-2, Optimus enables guided language generation from an abstract level using the latent vectors. Compared with BERT, Optimus can generalize better on low-resource language understanding tasks due to the smooth latent space structure. Extensive experimental results on a wide range of language tasks demonstrate the effectiveness of Optimus. It achieves new state-of-the-art on VAE language modeling benchmarks. We hope that our first pre-trained big VAE language model itself and results can help the NLP community renew the interests of deep generative models in the era of large-scale pre-training, and make these principled methods more practical.

Motivation & Objective

• To develop a large-scale, deep variational autoencoder (VAE) for natural language that learns a universal, structured latent space for sentences.
• To enable controlled, guided text generation by manipulating latent vectors, overcoming the lack of high-level control in autoregressive models like GPT-2.
• To improve zero-shot and few-shot performance on low-resource language understanding tasks by learning a smoother, more generalizable latent space than BERT.
• To demonstrate that pre-training a deep VAE on massive text data mitigates the KL vanishing problem and improves representation quality.
• To unify language understanding and generation under a single framework by integrating BERT-like encoding and GPT-2-like decoding with a shared latent space.

Proposed method

• Pre-train a deep Transformer-based VAE on a large-scale text corpus using the standard VAE objective: reconstructing input sentences from their latent codes.
• Use a prior distribution (e.g., standard normal) over the latent space to regularize the learned representations and ensure smoothness.
• Inject conditioning vectors into the decoder during inference, enabling guided text generation without retraining, using latent vector injection techniques.
• Fine-tune the pre-trained model on downstream tasks such as text generation, style transfer, and zero-shot understanding.
• Leverage the disentangled and structured nature of the latent space to perform arithmetic operations (e.g., vector addition) for semantic transfer.
• Combine the encoder of BERT and the decoder of GPT-2 into a unified architecture, reusing existing pre-trained components to build a scalable and effective model.

Experimental results

Research questions

• RQ1 Can a large-scale, deep VAE learn a universal and structured latent space for natural language that supports both generation and understanding?
• RQ2 How does pre-training on massive text data mitigate the KL vanishing problem in language VAEs?
• RQ3 Can latent vector arithmetic enable meaningful, interpretable sentence manipulation (e.g., style transfer, content transition)?
• RQ4 Does the structured latent space of Optimus generalize better than BERT’s representation on low-resource language understanding tasks?
• RQ5 Can Optimus achieve better performance than GPT-2 in language modeling while enabling controllable generation through latent codes?

Key findings

• Optimus achieves state-of-the-art performance on four standard VAE language modeling benchmarks, outperforming previous VAE-based models.
• Optimus achieves lower perplexity than GPT-2 on standard language modeling benchmarks, demonstrating superior sequence modeling capability.
• The model enables effective guided text generation via latent vector arithmetic, such as transferring sentiment, topic, or subject count from one sentence to another.
• On zero-shot and few-shot language understanding tasks, Optimus generalizes better than BERT due to the smoother, more structured latent space.
• The latent space supports meaningful sentence transfer through vector arithmetic, such as transforming a singular to plural subject or shifting from daily life to sports context.
• The proof of VAE objective from the information bottleneck lens confirms that VAE is a principled method to balance compactness and usability of learned representations.