[Paper Review] APo-VAE: Text Generation in Hyperbolic Space

Shuyang Dai, Zhe Gan+4 more|arXiv (Cornell University)|Apr 30, 2020

Topic Modeling46 references4 citations

TL;DR

APo-VAE proposes a hyperbolic variational autoencoder using wrapped normal distributions on a Poincaré ball to model hierarchical structures in natural language. By leveraging a primal-dual formulation of KL divergence for adversarial training, it achieves superior performance in text generation, language modeling, and style transfer while mitigating posterior collapse.

ABSTRACT

Natural language often exhibits inherent hierarchical structure ingrained with complex syntax and semantics. However, most state-of-the-art deep generative models learn embeddings only in Euclidean vector space, without accounting for this structural property of language. In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adversarial Poincare Variational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. By adopting the primal-dual formulation of KL divergence, an adversarial learning procedure is introduced to empower robust model training. Extensive experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model over VAEs in Euclidean latent space, thanks to its superb capabilities in capturing latent language hierarchies in hyperbolic space.

Motivation & Objective

• To address the limitation of Euclidean VAEs in capturing the inherent hierarchical structure of natural language.
• To explore whether hyperbolic geometry, particularly the Poincaré ball model, can better represent tree-structured linguistic hierarchies.
• To improve training stability and representation quality in VAE-based text generation by introducing adversarial optimization via primal-dual KL formulation.
• To mitigate posterior collapse in text VAEs by employing a more informative, data-driven prior in hyperbolic space.
• To demonstrate the effectiveness of hyperbolic latent spaces across multiple NLP tasks, including language modeling, style transfer, and dialogue generation.

Proposed method

• Defines both prior and variational posterior distributions over a Poincaré ball using wrapped normal distributions to model hierarchical latent structures.
• Employs a primal-dual formulation of the Kullback-Leibler divergence to enable adversarial optimization of the variational bound, avoiding reliance on Monte Carlo approximations.
• Introduces a VampPrior design with a mixture of variational posteriors to enhance prior complexity and improve representation learning.
• Applies adversarial training to disentangle content and sentiment in unaligned style transfer, using a sentiment classifier.
• Develops a conditional variant of APo-VAE for dialog-response generation, conditioning on dialog context.
• Uses two-dimensional projections of latent codes to visualize hierarchical structure, confirming that longer, more complex sentences occupy the outer rim of the Poincaré ball.

Experimental results

Research questions

• RQ1 Can hyperbolic latent spaces better capture the hierarchical structure of natural language compared to Euclidean spaces?
• RQ2 Does the use of wrapped normal distributions on the Poincaré ball enable more effective and stable training of VAEs for text generation?
• RQ3 Can the primal-dual KL formulation improve training stability and representation quality in text VAEs compared to standard variational inference?
• RQ4 To what extent does APo-VAE mitigate posterior collapse in text generation tasks?
• RQ5 How does APo-VAE perform across diverse NLP tasks such as language modeling, unaligned style transfer, and dialogue generation compared to existing VAE-based models?

Key findings

• APo-VAE achieves lower negative ELBO (116.2) and PPL (6.1) on the PTB dataset compared to all baselines, including iVAE, β-VAE, and P-VAE.
• The model exhibits a higher KL divergence term and greater mutual information between latent codes and inputs, indicating reduced posterior collapse.
• In unaligned style transfer, APo-VAE achieves a BLEU score of 37.8, outperforming iVAE (36.7), with significant gains in informativeness (52.8% win rate) and diversity (51.2% win rate).
• For dialogue response generation, APo-VAE improves average recall from 0.427 (iVAE) to 0.438 and increases intra-dist-2 from 0.692 to 0.792, indicating enhanced diversity.
• Human evaluation shows APo-VAE is preferred in 52.8% of cases for informativeness, 63.7% for coherence, and 62.1% for diversity over competing models.
• Visualization confirms that longer, more complex sentences are embedded toward the outer rim of the Poincaré ball, supporting the model’s ability to learn hierarchical representations.