[Paper Review] A Distributional Lens for Multi-Aspect Controllable Text Generation

Yuxuan Gu, Xiaocheng Feng+4 more|arXiv (Cornell University)|Oct 6, 2022

Topic Modeling4 citations

TL;DR

This paper proposes a distributional approach to multi-aspect controllable text generation by directly searching for the intersection of attribute distributions in a learned latent space, using an autoencoder and iterative optimization to find combined attribute regions. It outperforms strong baselines on sentiment, topic, and detoxification control, achieving SOTA results in attribute relevance and text quality.

ABSTRACT

Multi-aspect controllable text generation is a more challenging and practical task than single-aspect control. Existing methods achieve complex multi-aspect control by fusing multiple controllers learned from single-aspect, but suffer from attribute degeneration caused by the mutual interference of these controllers. To address this, we provide observations on attribute fusion from a distributional perspective and propose to directly search for the intersection areas of multiple attribute distributions as their combination for generation. Our method first estimates the attribute space with an autoencoder structure. Afterward, we iteratively approach the intersections by jointly minimizing distances to points representing different attributes. Finally, we map them to attribute-relevant sentences with a prefix-tuning-based decoder. Experiments on the three-aspect control task, including sentiment, topic, and detoxification aspects, reveal that our method outperforms several strong baselines on attribute relevance and text quality and achieves the SOTA. Further analysis also supplies some explanatory support for the effectiveness of our approach.

Motivation & Objective

• To address attribute degeneration in multi-aspect text generation caused by controller interference in existing fusion methods.
• To model complex, non-symmetric attribute distributions in a latent space that better reflect real-world data patterns.
• To directly search for the intersection of multiple attribute distributions as the target generation point, rather than relying on interpolation of distribution centers.
• To improve attribute relevance and text quality in multi-aspect control by leveraging a prefix-tuning-based decoder on the discovered intersection.
• To provide empirical and analytical support for the effectiveness of a distributional lens in multi-aspect text generation.

Proposed method

• An autoencoder structure is used to map attribute-relevant sentences into a shared latent space, estimating the attribute distribution.
• A novel intersection search algorithm iteratively minimizes distances to representative points of each attribute, navigating long-tail regions to locate tight intersections.
• The aspect gap loss is introduced to reduce inter-aspect distributional divergence, improving alignment in the latent space.
• A prefix-tuning-based decoder generates fluent, attribute-compliant sentences from the searched intersection points in the latent space.
• The method jointly optimizes reconstruction loss and aspect gap loss to preserve semantic fidelity and distributional coherence.
• 2D projections via PCA visualize the non-convex and skewed nature of attribute distributions, supporting the need for direct intersection search.

Experimental results

Research questions

• RQ1 Can a distributional lens improve multi-aspect text generation by modeling intersections of attribute distributions rather than interpolating centers?
• RQ2 How do real-world attribute distributions in text differ from symmetric assumptions like Gaussian distributions?
• RQ3 Does direct intersection search in the latent space lead to better attribute relevance and text quality than existing fusion strategies?
• RQ4 To what extent do non-convex and long-tailed attribute distributions challenge standard interpolation-based fusion methods?
• RQ5 How does the proposed method handle mutual interference between controllers in multi-aspect settings?

Key findings

• The proposed method achieves SOTA performance on a three-aspect control task involving sentiment, topic, and detoxification attributes.
• Experiments show significant improvements in attribute relevance and text quality compared to strong baselines, with lower perplexity and higher distinctness scores.
• 2D PCA projections of the latent space reveal that attribute distributions are non-convex and skewed, especially with long tails, supporting the need for intersection-based search.
• The method effectively reduces attribute degeneration by avoiding interpolation of distribution centers, which can lie outside true intersection regions.
• Analytical results confirm that the distributional assumptions underlying the method align well with observed data patterns in the latent space.
• The approach is robust to complex attribute interactions and maintains high fluency and diversity in generated text.