[Paper Review] Hierarchical Multiscale Recurrent Neural Networks

Jun‐Young Chung, Sungjin Ahn+1 more|arXiv (Cornell University)|Sep 6, 2016

Topic Modeling58 references249 citations

TL;DR

Introduces HM-RNN (HM-LSTM) that learns latent hierarchical multiscale structure in sequences without explicit boundaries, using adaptive boundary detectors and three operations (UPDATE, COPY, FLUSH). Demonstrates state-of-the-art results on character-level language modeling and strong performance on handwriting sequence generation.

ABSTRACT

Learning both hierarchical and temporal representation has been among the long-standing challenges of recurrent neural networks. Multiscale recurrent neural networks have been considered as a promising approach to resolve this issue, yet there has been a lack of empirical evidence showing that this type of models can actually capture the temporal dependencies by discovering the latent hierarchical structure of the sequence. In this paper, we propose a novel multiscale approach, called the hierarchical multiscale recurrent neural networks, which can capture the latent hierarchical structure in the sequence by encoding the temporal dependencies with different timescales using a novel update mechanism. We show some evidence that our proposed multiscale architecture can discover underlying hierarchical structure in the sequences without using explicit boundary information. We evaluate our proposed model on character-level language modelling and handwriting sequence modelling.

Motivation & Objective

• Motivate and address the challenge of jointly learning hierarchical and temporal representations in RNNs.
• Develop a model that discovers latent hierarchical structure without explicit boundary information.
• Improve efficiency and long-term dependency modeling through adaptive, multiscale updates.

Proposed method

• Proposes HM-RNN using a boundary detector at each layer that turns on to mark segment boundaries of a given abstraction level.
• Implements three operations per layer at each time step: UPDATE (sparse, when a boundary is detected), COPY (preserve previous state), FLUSH (eject segment representation to upper layer and reset).
• Extends to HM-LSTM by incorporating LSTM-style states and gates, with top-down and bottom-up interactions and a hard boundary signal.
• Uses straight-through estimator to train discrete boundary decisions and employs a slope annealing trick to reduce bias during training.
• Defines a training objective based on negative log-likelihood for sequence modeling and applies to character-level language modeling and handwriting sequence generation.
• Outputs are combined via layer-wise gates to form the final next-symbol distribution.

Experimental results

Research questions

• RQ1 Can a recurrent network discover latent hierarchical structure in sequences without explicit boundary labels?
• RQ2 How effectively can adaptive, multiscale updates capture temporal dependencies and reduce computational burden?
• RQ3 Do hierarchical boundary detectors align with natural linguistic or semantic boundaries in text data?
• RQ4 Is the straight-through estimator (with slope annealing) effective for training models with discrete boundary decisions?

Key findings

• HM-RNN discovered latent hierarchical structure in sequences without boundary supervision, with lower layers learning finer timescales and higher layers learning coarser timescales.
• On Penn Treebank, HM-LSTM with step boundary and slope annealing achieved 1.24 bits-per-character (BPC), competitive with or better than several baselines.
• On Text8, HM-LSTM achieved 1.29 BPC, the state-of-the-art among reported neural models at the time.
• On Hutter Prize Wikipedia, HM-LSTM reached 1.32 BPC, tying for the state-of-the-art neural result.
• Visualization showed boundary detectors align with plausible word/phrase boundaries and informative hierarchical segmentation.
• In handwriting sequence generation (IAM-OnDB), HM-LSTM outperformed standard LSTM in log-likelihood, demonstrating generalization to real-valued sequences.