[Paper Review] Plug and Play Autoencoders for Conditional Text Generation

Florian Mai, Nikolaos Pappas+3 more|arXiv (Cornell University)|Oct 6, 2020

Topic Modeling51 references4 citations

TL;DR

This paper introduces Emb2Emb, a plug-and-play framework that enables conditional text generation by learning a mapping in the latent space of any pretrained text autoencoder, using adversarial loss and offset-based neural mapping. It achieves state-of-the-art or competitive performance on style transfer tasks with up to four times faster training than end-to-end models, while requiring minimal labeled data and preserving content better in unsupervised settings.

ABSTRACT

Text autoencoders are commonly used for conditional generation tasks such as style transfer. We propose methods which are plug and play, where any pretrained autoencoder can be used, and only require learning a mapping within the autoencoder's embedding space, training embedding-to-embedding (Emb2Emb). This reduces the need for labeled training data for the task and makes the training procedure more efficient. Crucial to the success of this method is a loss term for keeping the mapped embedding on the manifold of the autoencoder and a mapping which is trained to navigate the manifold by learning offset vectors. Evaluations on style transfer tasks both with and without sequence-to-sequence supervision show that our method performs better than or comparable to strong baselines while being up to four times faster.

Motivation & Objective

• To enable conditional text generation with minimal labeled data by leveraging pretrained autoencoders.
• To develop a plug-and-play method that works with any existing autoencoder without retraining.
• To improve efficiency and content preservation in unsupervised style transfer tasks.
• To maintain generation quality by ensuring mapped embeddings remain on the autoencoder's latent manifold.
• To reduce training cost and dependency on parallel data in sequence-to-sequence generation.

Proposed method

• The method learns a mapping Φ from input sentence embeddings to target sentence embeddings within the autoencoder’s latent space.
• It uses an adversarial loss term ℒ_adv to enforce that the output embedding lies on the autoencoder’s manifold, ensuring valid generation.
• The mapping network is designed to learn and apply offset vectors to the input embedding, enabling fine-grained adjustments.
• For supervised learning, the method minimizes a task loss ℒ_task between predicted and true target embeddings.
• The framework is compatible with any pretrained autoencoder, regardless of whether it was trained with style attributes in mind.
• At inference, the learned mapping is composed between the autoencoder’s encoder and decoder to generate output text.

Experimental results

Research questions

• RQ1 Can a pretrained autoencoder be used for conditional text generation without retraining or fine-tuning?
• RQ2 How can a mapping in the latent space improve content preservation and generation quality in style transfer?
• RQ3 Can adversarial regularization in the embedding space maintain validity and improve performance?
• RQ4 How does the offset-based mapping architecture compare to end-to-end training in terms of speed and accuracy?
• RQ5 To what extent can pretraining on unlabeled data improve performance in low-resource conditional generation settings?

Key findings

• Emb2Emb outperforms strong baselines in text simplification, a supervised style transfer task, while being up to four times faster than end-to-end models.
• In unsupervised sentiment transfer, Emb2Emb preserves input content better than a state-of-the-art method while achieving comparable transfer accuracy.
• The method achieves performance close to an upper bound model that uses ten times more labeled data, demonstrating strong low-resource generalization.
• The adversarial loss term ℒ_adv is crucial for maintaining embeddings on the autoencoder manifold, directly improving generation quality.
• The offset-based mapping architecture enables effective, stable adjustments to input embeddings, improving task-specific adaptation.
• The plug-and-play nature of Emb2Emb allows seamless integration with any pretrained autoencoder, enabling reuse of large-scale pretraining.