[Paper Review] Stochastic Sequential Neural Networks with Structured Inference

Hao Liu, Haoli Bai+2 more|arXiv (Cornell University)|May 24, 2017

Music and Audio Processing40 references3 citations

TL;DR

This paper proposes a Stochastic Sequential Neural Network (SSNN) that combines recurrent neural networks with structured inference to model long-term dependencies and discrete segmentation/labeling in high-dimensional sequential data. By using Gumbel-Softmax reparameterization and bidirectional inference, SSNN enables efficient, differentiable training of discrete latent variables, achieving state-of-the-art performance in unsupervised segmentation, labeling, and multi-object recognition tasks.

ABSTRACT

Unsupervised structure learning in high-dimensional time series data has attracted a lot of research interests. For example, segmenting and labelling high dimensional time series can be helpful in behavior understanding and medical diagnosis. Recent advances in generative sequential modeling have suggested to combine recurrent neural networks with state space models (e.g., Hidden Markov Models). This combination can model not only the long term dependency in sequential data, but also the uncertainty included in the hidden states. Inheriting these advantages of stochastic neural sequential models, we propose a structured and stochastic sequential neural network, which models both the long-term dependencies via recurrent neural networks and the uncertainty in the segmentation and labels via discrete random variables. For accurate and efficient inference, we present a bi-directional inference network by reparamterizing the categorical segmentation and labels with the recent proposed Gumbel-Softmax approximation and resort to the Stochastic Gradient Variational Bayes. We evaluate the proposed model in a number of tasks, including speech modeling, automatic segmentation and labeling in behavior understanding, and sequential multi-objects recognition. Experimental results have demonstrated that our proposed model can achieve significant improvement over the state-of-the-art methods.

Motivation & Objective

• To address the challenge of unsupervised structure learning in high-dimensional sequential data, such as segment boundaries and labels in physiological signals or behavior sequences.
• To enable effective modeling of both long-term dependencies and discrete, interpretable latent structures (segmentation and labels) in sequential data.
• To overcome the difficulty of training deep neural networks with discrete latent variables by introducing a structured, bidirectional inference mechanism.
• To improve inference efficiency and accuracy in sequential models with categorical latent variables through Gumbel-Softmax reparameterization.
• To demonstrate the effectiveness of the proposed model across diverse tasks including speech modeling, behavior understanding, medical diagnosis, and multi-object recognition.

Proposed method

• The model uses a generative network inspired by Hidden Semi-Markov Models (HSMMs), with continuous hidden states from an RNN and two discrete sequences: segmentation variables and segment labels.
• Discrete latent variables (segmentation and labels) are reparameterized using the Gumbel-Softmax trick to enable differentiable training despite their categorical nature.
• A bidirectional inference network is designed to leverage both forward and backward temporal context, improving posterior approximation of discrete latent variables.
• Stochastic Gradient Variational Bayes (SGVB) is employed for end-to-end training, optimizing the variational lower bound (ELBO) with reparameterized gradients.
• The model jointly learns the duration of segments and their labels, with inference guided by both past and future context through augmented hidden states.
• The architecture is trained end-to-end using mini-batch SGD with a fixed maximum sequence length and truncation of duration to prevent computational blowup.

Experimental results

Research questions

• RQ1 Can a deep neural network effectively learn interpretable segment boundaries and labels in high-dimensional sequential data without explicit supervision?
• RQ2 How can discrete latent variables (segmentation and labels) be efficiently optimized in a differentiable way within a deep learning framework?
• RQ3 Does incorporating bidirectional temporal context in the inference network improve the accuracy of structured inference for sequential data?
• RQ4 Can the proposed model outperform existing state-of-the-art methods in unsupervised sequential structure learning across diverse benchmarks?
• RQ5 How well does the model generalize to complex tasks involving spatial and sequential dependencies, such as multi-object recognition in images?

Key findings

• SSNN achieves the lowest mean error rate across all evaluated datasets, with 14.70 ± 5.45% on the Drosophila dataset and 29.29 ± 5.34% on the PhysioNet dataset, outperforming subHSMM, HDP-HSMM, CRF-AE, and rHSMM-dp.
• On the multi-MNIST dataset, SSNN successfully identifies the number and locations of three digits in images, while DRAW sometimes misses data modes, indicating superior mode coverage and disentanglement.
• The bidirectional inference network with Gumbel-Softmax reparameterization enables accurate and stable training of discrete latent variables, avoiding the gradient issues common in discrete variational inference.
• The model demonstrates strong generalization in unsupervised settings, learning meaningful segmentations and labels from raw sequences without supervision or curriculum learning.
• In speech and behavior modeling tasks, SSNN achieves significant improvements in segmentation and labeling accuracy, confirming its effectiveness in real-world sequential data analysis.
• The ablation study confirms that the bidirectional inference and structured modeling of discrete variables are key to the model’s performance, with ablation leading to notable performance drops.