[Paper Review] Spectral dimensionality reduction for HMMs

Dean P. Foster, Jordan Rodu+1 more|arXiv (Cornell University)|Mar 28, 2012

Bayesian Methods and Mixture Models11 references20 citations

TL;DR

This paper proposes a spectral dimensionality reduction method for Hidden Markov Models (HMMs) that replaces the high-parameter tensor $B_x$ of Hsu et al. (2009) with a lower-dimensional tensor $C(y)$ depending on a projected observation $y = U^T \delta_x$, reducing model parameters from $m^2v$ to $m^3$. The method achieves sample complexity independent of vocabulary size $v$, with theoretical bounds on relative error that depend only on checkable data conditions and the smallest singular value $\sigma_m$ of the covariance matrix.

ABSTRACT

Hidden Markov Models (HMMs) can be accurately approximated using co-occurrence frequencies of pairs and triples of observations by using a fast spectral method in contrast to the usual slow methods like EM or Gibbs sampling. We provide a new spectral method which significantly reduces the number of model parameters that need to be estimated, and generates a sample complexity that does not depend on the size of the observation vocabulary. We present an elementary proof giving bounds on the relative accuracy of probability estimates from our model. (Correlaries show our bounds can be weakened to provide either L1 bounds or KL bounds which provide easier direct comparisons to previous work.) Our theorem uses conditions that are checkable from the data, instead of putting conditions on the unobservable Markov transition matrix.

Motivation & Objective

• To reduce the number of model parameters in spectral HMM estimation, especially when the observation vocabulary $v$ is large.
• To eliminate dependence of sample complexity on the size of the observation vocabulary $v$.
• To develop a method with theoretical bounds on relative error that depend only on observable data conditions, not unobservable transition matrices.
• To simplify and strengthen the theoretical analysis of spectral HMM learning by using only checkable data-based conditions.

Proposed method

• Project high-dimensional observations $x$ into a low-dimensional space using $y = U^T \delta_x$, where $U$ is derived from the SVD of bigram co-occurrence matrices.
• Replace the original $m^2v$-parameter tensor $B_x$ with a reduced $m^3$-parameter tensor $C(y)$ that operates on the projected observation $y$.
• Estimate the sufficient statistics $\mu = E(y_1)$, $\Sigma = E(y_2 y_1^T)$, and $K(a) = E(y_3 y_1^T y_2^T a)$ using method-of-moments on trigram frequencies.
• Use the estimated $c_1 = \mu$, $c_\infty^\top = \mu^\top \Sigma^{-1}$, and $C(y) = K(y) \Sigma^{-1}$ to compute sequence probabilities via recursive matrix products.
• Improve numerical stability by estimating likelihood ratios $\lambda_q(x_{1:t}) = \frac{Pr(x_{1:t})}{P_1(x_1)\cdots P_1(x_t)}$ instead of raw probabilities.
• Bound relative error using data-checkable conditions on $\Sigma$ and $\sigma_m$, the smallest singular value of $\Sigma$, rather than on unobservable transition matrices.

Experimental results

Research questions

• RQ1 Can spectral HMM estimation be made more efficient by reducing the number of parameters from $m^2v$ to $m^3$?
• RQ2 Does the sample complexity of spectral HMM learning become independent of the observation vocabulary size $v$ when using dimensionality reduction?
• RQ3 Can theoretical error bounds be derived using only checkable data conditions, rather than assumptions on unobservable transition matrices?
• RQ4 How does the smallest singular value $\sigma_m$ of the covariance matrix $\Sigma$ affect the identifiability and sample complexity of the HMM?
• RQ5 Can numerical instability in probability estimation be mitigated by working with likelihood ratios instead of raw probabilities?

Key findings

• The proposed method reduces the number of parameters to estimate from $m^2v$ to $m^3$, achieving a significant reduction when $v \gg m$.
• The sample complexity of the model depends critically on $\sigma_m$, the smallest singular value of the covariance matrix $\Sigma$, with smaller $\sigma_m$ requiring more samples.
• Empirical results show that $\Lambda$, the smallest parameter in the model, decreases with a power law of $m$ (slope ≈ -6), and $\widehat{\sigma}_m$ decreases with slope ≈ -3.2, indicating increasing difficulty in identification as $m$ grows.
• Theoretical bounds on relative error are derived using only data-checkable conditions on $\Sigma$ and $\sigma_m$, improving upon prior work that relied on unobservable transition matrix assumptions.
• Using likelihood ratios instead of raw probabilities improves numerical stability and enhances relative accuracy, especially in high-dimensional settings.
• The method maintains theoretical guarantees with simpler proofs and provides L1 and KL bounds as corollaries, enabling direct comparison with previous approaches.