[Paper Review] Stochastic WaveNet: A Generative Latent Variable Model for Sequential Data

Guokun Lai, Bohan Li+2 more|arXiv (Cornell University)|Jun 15, 2018

Time Series Analysis and ForecastingComputer Science24 references19 citations

TL;DR

This paper proposes Stochastic WaveNet, a generative latent variable model that integrates stochastic latent variables into the dilated convolutional architecture of WaveNet to enhance distribution modeling capacity for sequential data. By leveraging parallel training via dilated convolutions and a specialized inference network, the model achieves state-of-the-art log-likelihood performance on speech and handwriting generation benchmarks, outperforming both vanilla WaveNet and RNN-based stochastic models.

ABSTRACT

How to model distribution of sequential data, including but not limited to speech and human motions, is an important ongoing research problem. It has been demonstrated that model capacity can be significantly enhanced by introducing stochastic latent variables in the hidden states of recurrent neural networks. Simultaneously, WaveNet, equipped with dilated convolutions, achieves astonishing empirical performance in natural speech generation task. In this paper, we combine the ideas from both stochastic latent variables and dilated convolutions, and propose a new architecture to model sequential data, termed as Stochastic WaveNet, where stochastic latent variables are injected into the WaveNet structure. We argue that Stochastic WaveNet enjoys powerful distribution modeling capacity and the advantage of parallel training from dilated convolutions. In order to efficiently infer the posterior distribution of the latent variables, a novel inference network structure is designed based on the characteristics of WaveNet architecture. State-of-the-art performances on benchmark datasets are obtained by Stochastic WaveNet on natural speech modeling and high quality human handwriting samples can be generated as well.

Motivation & Objective

• To enhance the representational capacity of WaveNet for sequential data by introducing stochastic latent variables in hidden states.
• To enable efficient, parallel training by adapting the dilated convolutional architecture of WaveNet to stochastic modeling.
• To design a dedicated inference network that effectively captures posterior dependencies in the latent space, leveraging WaveNet's receptive field structure.
• To empirically validate that multi-layer stochastic latent variables improve modeling of hierarchical and sequential data structures.
• To demonstrate state-of-the-art performance on benchmark datasets for speech and handwriting generation.

Proposed method

• Inject stochastic latent variables into every hidden layer of the WaveNet architecture, enabling uncertainty modeling across multiple hierarchical levels.
• Use dilated convolutions to maintain long-range temporal context and enable parallel training, unlike autoregressive RNNs.
• Design a conditional inference network that estimates the posterior distribution of latent variables using the entire observed sequence, leveraging the same dilated convolutional structure.
• Optimize the variational lower bound (ELBO) to maximize data log-likelihood, jointly training the generative and inference networks.
• Control the number of stochastic layers and latent dimensions to study their impact on model performance and hierarchical representation learning.
• Apply the model to sequential data such as speech waveforms and handwriting trajectories, using discrete or continuous latent variables depending on the data modality.

Experimental results

Research questions

• RQ1 Can injecting stochastic latent variables into the hidden states of WaveNet improve its capacity to model complex, multimodal data distributions?
• RQ2 Does the use of multi-layer stochastic latent variables in a dilated convolutional architecture enable better hierarchical representation learning than single-layer stochastic RNNs?
• RQ3 Can the proposed inference network effectively approximate the true posterior of latent variables in a non-autoregressive, parallelizable setting?
• RQ4 How does the number of stochastic layers and latent dimensions affect the model's log-likelihood and sample quality on sequential data?
• RQ5 Does Stochastic WaveNet achieve state-of-the-art performance on benchmark sequential generation tasks like speech and handwriting?

Key findings

• Stochastic WaveNet achieves state-of-the-art log-likelihood on the Blizzard and TIMIT speech datasets, with results comparable to the best-performing models.
• On the IAM-OnDB handwriting dataset, Stochastic WaveNet achieves a log-likelihood of at least 1301, outperforming RNN-based models and approaching the best reported result of 1334.
• The model generates higher-quality handwriting samples than RNN and VRNN, with clearer character boundaries and more realistic character shapes such as 'is' and 'her'.
• Empirical results show that increasing the number of stochastic layers improves performance up to a point, indicating that hierarchical latent structures enhance modeling capacity.
• Performance degrades when the number of stochastic layers becomes too large, as each layer receives too few latent variables to effectively capture meaningful representations.
• Even a small number of stochastic latent variables significantly improves generation quality, demonstrating the model's efficiency in leveraging uncertainty for better distribution modeling.