[Paper Review] Improving Disentangled Text Representation Learning with Information-Theoretic Guidance

Pengyu Cheng, Martin Renqiang Min+5 more|arXiv (Cornell University)|Jun 1, 2020

Topic Modeling38 references4 citations

TL;DR

This paper proposes IDEL, a novel information-theoretic framework for learning disentangled text representations by minimizing a sample-based mutual information upper bound to reduce dependency between style and content embeddings, while maximizing mutual information with input text to preserve semantic content. The method achieves state-of-the-art performance in style transfer and conditional generation with strong disentanglement and content preservation.

ABSTRACT

Learning disentangled representations of natural language is essential for many NLP tasks, e.g., conditional text generation, style transfer, personalized dialogue systems, etc. Similar problems have been studied extensively for other forms of data, such as images and videos. However, the discrete nature of natural language makes the disentangling of textual representations more challenging (e.g., the manipulation over the data space cannot be easily achieved). Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text, without any supervision on semantics. A new mutual information upper bound is derived and leveraged to measure dependence between style and content. By minimizing this upper bound, the proposed method induces style and content embeddings into two independent low-dimensional spaces. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation in terms of content and style preservation.

Motivation & Objective

• To address the challenge of learning disentangled representations in natural language, where the discrete nature of text complicates manipulation and disentanglement.
• To develop a self-supervised method that disentangles style and content without requiring manual annotations or predefined style categories.
• To provide a theoretically grounded objective based on information theory that ensures low dependency between style and content embeddings.
• To preserve semantic content in latent representations by maximizing mutual information between input sentences and latent codes.
• To achieve a balanced trade-off between style transfer accuracy and content preservation in downstream tasks.

Proposed method

• Proposes a novel information-theoretic objective based on Variation of Information (VI) to measure and minimize the dependence between style and content embeddings.
• Derives a sample-based mutual information upper bound to enable stable and effective minimization of embedding correlation during training.
• Integrates the upper bound minimization with a reconstruction objective to ensure that latent representations retain sufficient information about the input sentence.
• Maximizes mutual information between input text and latent representations to preserve content fidelity.
• Employs a variational autoencoder-based architecture with separate encoders for style and content, trained end-to-end using the proposed objective.
• Uses a stochastic approximation of the mutual information upper bound to improve training efficiency and performance.

Experimental results

Research questions

• RQ1 Can an information-theoretic objective effectively disentangle style and content representations in natural language without supervision?
• RQ2 How can mutual information between style and content embeddings be minimized in a differentiable and stable way for discrete text data?
• RQ3 Does minimizing embedding dependency while preserving input information lead to better performance in style transfer and conditional generation?
• RQ4 What is the trade-off between style transfer accuracy and content preservation, and can the proposed method achieve a better balance?
• RQ5 Can the model generalize to unseen style classes without fine-tuning, given its self-supervised nature?

Key findings

• IDEL achieves a geometric mean (GM) score of 41.9 on the Yelp dataset for style transfer, outperforming strong baselines like BT and ARAE.
• On human evaluation, IDEL achieves a style accuracy of 73.7% and content preservation of 3.69/5, outperforming CtrlGen, CAAE, and ARAE.
• The ablation study shows that removing the mutual information upper bound (IDEL -) leads to a significant drop in style transfer accuracy, confirming its critical role.
• Adding the mutual information term I(s;y) improves style transfer accuracy but harms content preservation, while I(c;x) improves content fidelity at the cost of style accuracy, demonstrating the trade-off.
• The stochastic variant (IDEL*) outperforms the closed-form version (IDEL∗), indicating that stochastic estimation enhances both training speed and model performance.
• IDEL achieves high BLEU and S-BLEU scores while maintaining strong style transfer performance, indicating a balanced and high-quality disentangled representation.