[Paper Review] Learning Overcomplete HMMs

Vatsal Sharan, Sham M. Kakade+2 more|arXiv (Cornell University)|Nov 7, 2017

Machine Learning and Algorithms18 references8 citations

TL;DR

This paper investigates the learnability of overcomplete Hidden Markov Models (HMMs), where the number of hidden states greatly exceeds the output alphabet size. It establishes a fundamental information-theoretic lower bound showing that HMMs with random regular transition matrices and small output alphols cannot be learned from polynomially many samples, while proving positive results for sparse HMMs with limited short-cycle probability using tensor decomposition and the method of moments under random observation matrices.

ABSTRACT

We study the problem of learning overcomplete HMMs---those that have many hidden states but a small output alphabet. Despite having significant practical importance, such HMMs are poorly understood with no known positive or negative results for efficient learning. In this paper, we present several new results---both positive and negative---which help define the boundaries between the tractable and intractable settings. Specifically, we show positive results for a large subclass of HMMs whose transition matrices are sparse, well-conditioned, and have small probability mass on short cycles. On the other hand, we show that learning is impossible given only a polynomial number of samples for HMMs with a small output alphabet and whose transition matrices are random regular graphs with large degree. We also discuss these results in the context of learning HMMs which can capture long-term dependencies.

Motivation & Objective

• To understand the fundamental limits of learning overcomplete HMMs, where the number of hidden states n greatly exceeds the output alphabet size m.
• To identify structural properties of transition matrices that enable efficient learning with polynomial sample complexity.
• To resolve the tension between identifiability and sample complexity in overcomplete HMMs, especially when standard spectral methods fail.
• To clarify whether generic or random HMMs are learnable in the overcomplete regime, given that they may be identifiable yet information-theoretically intractable.
• To develop a framework for analyzing long-term dependency capture in HMMs through a new memory notion based on conditional state influence.

Proposed method

• Proves an information-theoretic lower bound using a counting argument on the number of random regular graphs with degree d = n^ε, showing that polynomial samples contain insufficient information to specify the transition matrix.
• Analyzes the decay of influence of the initial hidden state on future states, conditioned on observed outputs, by modeling the posterior distribution as a time-inhomogeneous Markov chain.
• Uses spectral analysis of time-inhomogeneous transition matrices to show that the influence of the initial state diminishes exponentially over time, even under conditioning.
• Proposes a positive learning algorithm for HMMs with sparse transition matrices and low probability mass on cycles shorter than 10 log_m n, leveraging tensor decomposition and the method of moments.
• Assumes random observation matrices with i.i.d. entries and E[O_ij] = 1/m to decouple the complexity of the observation matrix from the transition matrix structure.
• Introduces a new measure of memory in HMM output processes based on the conditional influence of past states on future predictions, which helps characterize long-term dependency capture.

Experimental results

Research questions

• RQ1 Can overcomplete HMMs with small output alphabet and large number of hidden states be learned efficiently with polynomial sample complexity?
• RQ2 What structural properties of the transition matrix make learning tractable in the overcomplete regime?
• RQ3 Are there information-theoretic barriers to learning HMMs with random regular transition matrices, even when they are identifiable?
• RQ4 How does the presence of short cycles in the transition graph affect the learnability of HMMs?
• RQ5 What is the relationship between the memory of the output process and the ability of an HMM to capture long-term dependencies?

Key findings

• HMMs with transition matrices corresponding to random walks on d-regular graphs with d = n^ε and m = polylog(n) cannot be learned from polynomially many samples, even approximately, due to information-theoretic limitations.
• The number of bits of information in polynomial-length windows is at most Õ(n), while Ω(nd) bits are needed to specify the transition matrix, making learning impossible.
• For sparse HMMs with bounded degree and low probability mass on cycles shorter than 10 log_m n, efficient learning is possible using tensor decomposition and the method of moments with random observation matrices.
• The condition on short cycles ensures sufficient mixing and exploration of the state space within polynomial time windows, enabling identifiability via higher-order moments.
• The influence of the initial hidden state on future states decays exponentially when conditioned on intermediate observations, which underpins the information-theoretic lower bound.
• The paper introduces a new notion of memory in HMMs based on the conditional influence of past states, which helps quantify long-term dependency capture and relates to learnability.