[Paper Review] WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling

Hao Zhang, Bo Chen+2 more|arXiv (Cornell University)|Mar 4, 2018

Generative Adversarial Networks and Image SynthesisComputer Science32 references51 citations

TL;DR

WHAI combines stochastic-gradient MCMC with a Weibull-based variational encoder to perform scalable, fast out-of-sample inference for deep topic models with a DLDA-based decoder.

ABSTRACT

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.

Motivation & Objective

• Motivate scalable inference for deep multilayer topic models that works well on large corpora.
• Develop an inference network that can efficiently approximate posteriors for sparse, nonnegative latent document representations.
• Integrate a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes to infer both global and local parameters.
• Leverage a Weibull-based encoder to approximate gamma-posteriors with analytic KL and simple reparameterization.
• Demonstrate improved performance and speed over competing inference strategies on large-scale text datasets.

Proposed method

• Use DLDA (a deep Poisson/gamma topic model) as the generative decoder to capture hierarchical document representations.
• Design an upward-downward inference network (WUDVE) where a bottom-up neural network feeds a stochastic downward generative model.
• Approximate gamma-conditionals with Weibull distributions in the encoder to enable analytic KL terms and reparameterizable sampling.
• Employ a hybrid inference algorithm that updates global parameters with TLASGR-MCMC and local parameters with the Weibull encoder, enabling fast yet accurate posterior inference.
• Compare variations including GHAI and WAI to isolate the impact of the stochastic-downward path and distribution choices.

Experimental results

Research questions

• RQ1 Does WHAI improve out-of-sample prediction and perplexity over existing deep topic models and variational autoencoders?
• RQ2 How does the hybrid MCMC/VAE framework perform in terms of scalability and speed on large corpora (training and testing) compared to Gibbs sampling and TLASGR-MCMC?
• RQ3 What is the impact of using Weibull (vs. gamma) in the inference network on posterior approximation and training efficiency?
• RQ4 Does the upward-downward information propagation in WHAI provide tangible benefits over traditional VAE or independent/incomplete top-down connections?
• RQ5 How do DHLA (WHAI) variants (with/without stochastic-downward paths) affect performance?

Key findings

• WHAI achieves favorable perplexity and test-time efficiency compared with state-of-the-art DLDA-based methods.
• The Weibull-based encoder closely approximates gamma posteriors with an analytic KL and easily reparameterizable sampling, enabling efficient gradient computation.
• The hybrid MCMC/VAE approach yields competitive or superior performance to Gibbs sampling while offering scalable mini-batch training for big corpora.
• In experiments, WHAI and its variants outperform their non-top-down counterparts, highlighting the benefit of stochastic-downward information flow.
• DLDA-WHAI with deeper architectures shows improved performance over shallower configurations, demonstrating effective hierarchical topic learning.
• Compared against AVITM and DPFA, models using a deep generative DLDA decoder consistently achieve better held-out perplexity while maintaining fast out-of-sample inference.