[Paper Review] Improving the Performance of the LSTM and HMM Model via Hybridization

Larkin Liu, Yu-Chung Lin+1 more|arXiv (Cornell University)|Jul 9, 2019

Topic Modeling4 citations

TL;DR

This paper proposes a hybrid modeling approach that leverages structural similarities between Hidden Markov Models (HMMs) and Long Short-Term Memory (LSTM) networks to improve language modeling performance. By analyzing hidden state probability distributions, the authors demonstrate that HMMs can effectively approximate LSTM behavior, especially at lower hidden state dimensions, leading to a potential simplification or enhancement of LSTM training through hybridization with HMMs.

ABSTRACT

Language models based on deep neural networks and traditional stochastic modelling have become both highly functional and effective in recent times. In this work, a general survey into the two types of language modelling is conducted. We investigate the effectiveness of the Hidden Markov Model (HMM), and the Long Short-Term Memory Model (LSTM). We analyze the hidden state structures common to both models, and present an analysis on structural similarity of the hidden states, common to both HMM's and LSTM's. We compare the LSTM's predictive accuracy and hidden state output with respect to the HMM for a varying number of hidden states. In this work, we justify that the less complex HMM can serve as an appropriate approximation of the LSTM model.

Motivation & Objective

• To investigate the structural and functional similarities between HMMs and LSTMs in modeling sequential language data.
• To evaluate whether the simpler HMM can serve as an effective approximation of the more complex LSTM model.
• To improve predictive accuracy and training stability by hybridizing HMM and LSTM architectures.
• To analyze the relationship between hidden state dimensionality and model performance in both HMM and LSTM frameworks.
• To explore the feasibility of using HMMs to simplify or enhance LSTM training through structural approximation.

Proposed method

• The authors compare the hidden state probability distributions of HMMs and LSTMs using cosine similarity on the same text sequences.
• They train an LSTM with varying numbers of hidden states (nh) and compare its forward pass output to the HMM’s hidden state probabilities.
• The HMM is trained using the Baum-Welch algorithm with forward and backward probabilities defined via Equations (2) and (3).
• Model performance is evaluated using cross-entropy loss J(θ), with log-likelihood ℒ(θ) = -J(θ) as the primary metric.
• The cosine distance Δ(ΨH, ΨL) between HMM and LSTM hidden state probability vectors is computed to quantify structural similarity.
• Experiments are conducted on multiple corpora, including the War and Peace (WP) dataset, with mini-batch SGD (m=64) for training and validation.

Experimental results

Research questions

• RQ1 To what extent do the hidden state probability distributions of HMMs and LSTMs exhibit structural similarity?
• RQ2 Can the HMM serve as a viable approximation of the LSTM in terms of predictive performance and hidden state dynamics?
• RQ3 How does the number of hidden states (nh) affect the cosine similarity between HMM and LSTM hidden states?
• RQ4 Does increasing the number of hidden states in the LSTM consistently improve predictive accuracy, or is there a saturation point?
• RQ5 Can hybridization of HMM and LSTM models lead to improved training stability or performance?

Key findings

• The cosine similarity between HMM and LSTM hidden state probability vectors (Δ(ΨH, ΨL)) increases in the 15–35 hidden state range for complex corpora like War and Peace, indicating stronger structural alignment at moderate dimensions.
• For the War and Peace dataset, the cross-entropy loss J(θ) decreased from 3.950 (nh=5) to 3.359 (nh=35), showing improved predictive accuracy with increased complexity.
• Despite increasing model complexity, the LSTM's performance does not monotonically improve; for example, J(θ) for WP increased slightly from 3.678 (nh=15) to 3.497 (nh=25), suggesting a non-linear relationship.
• The HMM outperforms the LSTM in terms of training speed when using a GPU, due to limited parallelization opportunities in small-layered networks.
• The authors observe that HMMs can effectively approximate LSTM behavior, especially at lower dimensionalities, supporting the feasibility of hybrid models.
• The study concludes that HMMs can be used as a suitable approximator for LSTMs, enabling potential simplification or performance gains in hybrid architectures.