[Paper Review] Learning Markov models via low-rank optimization

Ziwei Zhu, Xudong Li+2 more|arXiv (Cornell University)|Jun 28, 2019

Sparse and Compressive Sensing Techniques62 references4 citations

TL;DR

This paper proposes a low-rank optimization framework for learning Markov models by constructing a robust intermediate estimator $\\[\widehat{\mathbf{P}}_{\eta}\ $ that adaptively interpolates between the empirical estimator $ \widehat{\mathbf{P}}$ and the true transition matrix $ \mathbf{P}$. The key contribution is a non-asymptotic Frobenius norm error bound of order $O\left(\left(\frac{r p^2 \pi_{\max} \log p}{n \alpha}\right)^{1/2}\right)$, establishing optimal statistical rates under low-rank structure and sparsity assumptions.

ABSTRACT

Modeling unknown systems from data is a precursor of system optimization and sequential decision making. In this paper, we focus on learning a Markov model from a single trajectory of states. Suppose that the transition model has a small rank despite of having a large state space, meaning that the system admits a low-dimensional latent structure. We show that one can estimate the full transition model accurately using a trajectory of length that is proportional to the total number of states. We propose two maximum likelihood estimation methods: a convex approach with nuclear-norm regularization and a nonconvex approach with rank constraint. We explicitly derive the statistical rates of both estimators in terms of the Kullback-Leiber divergence and the $\ell_2$ error and also establish a minimax lower bound to assess the tightness of these rates. For computing the nonconvex estimator, we develop a novel DC (difference of convex function) programming algorithm that starts with the convex M-estimator and then successively refines the solution till convergence. Empirical experiments demonstrate consistent superiority of the nonconvex estimator over the convex one.

Motivation & Objective

• To develop a statistically optimal method for learning Markov transition matrices under low-rank and sparsity constraints.
• To address the challenge of high-dimensional estimation in Markov models where the number of states $p$ is large relative to sample size $n$.
• To improve estimation accuracy by introducing a robust intermediate estimator $\widehat{\mathbf{P}}_{\eta}$ that adapts to the unknown error size.
• To derive non-asymptotic error bounds for the Frobenius norm of the estimation error under minimal assumptions.
• To establish the optimality of the proposed method via comparison with known minimax lower bounds.

Proposed method

• Introduce an intermediate estimator $\widehat{\mathbf{P}}_{\eta} = \mathbf{P} + \eta(\widehat{\mathbf{P}} - \mathbf{P})$, where $\eta$ is chosen to truncate the estimation error based on a threshold $R$.
• Use a restricted strong convexity and gradient control argument to bound the estimation error via a dual norm and nuclear norm regularization.
• Apply a projection-based analysis using the normal cone $\Pi_N(\nabla L_n(\mathbf{P}))$ to control the subgradient term in the loss difference.
• Establish a chain of inequalities linking the loss difference to the nuclear norm of the error, ultimately bounding the Frobenius norm via low-rank structure.
• Derive a critical threshold $R_0 = \frac{C_4 \beta^2}{\alpha^2} \left(\frac{\xi r p^2 \pi_{\max} \log p}{n \alpha}\right)^{1/2}$ that ensures the error remains within the desired bound.
• Leverage results from Zhang (2018) to connect the Frobenius error to the Kullback-Leibler divergence, confirming statistical consistency.

Experimental results

Research questions

• RQ1 Can a low-rank optimization framework achieve optimal estimation rates for Markov transition matrices in high-dimensional settings?
• RQ2 How can an adaptive intermediate estimator $\widehat{\mathbf{P}}_{\eta}$ improve robustness to large estimation errors without prior knowledge of the true error magnitude?
• RQ3 What is the optimal trade-off between sample size $n$, dimension $p$, rank $r$, and sparsity $\pi_{\max}$ in achieving minimax-optimal error rates?
• RQ4 To what extent does the nuclear norm regularization improve estimation accuracy under low-rank constraints?
• RQ5 Does the proposed method achieve the known minimax lower bound for Markov model estimation?

Key findings

• The Frobenius norm error $\|\widehat{\mathbf{P}} - \mathbf{P}\|_F$ is bounded with high probability by $O\left(\left(\frac{r p^2 \pi_{\max} \log p}{n \alpha}\right)^{1/2}\right)$, matching the minimax optimal rate.
• The intermediate estimator $\widehat{\mathbf{P}}_{\eta}$ ensures that the estimation error remains within the threshold $R_0$, enabling tight control of the error norm.
• The method achieves optimal statistical performance under low-rank and sparse transition matrix assumptions, even when $p \gg n$.
• The Kullback-Leibler divergence between the estimated and true model is bounded by the same rate, confirming consistency in information-theoretic terms.
• The derived threshold $R_0$ is universal up to a constant and depends on the problem's intrinsic complexity: rank $r$, sparsity $\pi_{\max}$, and sample size $n$.
• The analysis confirms that the nuclear norm regularization effectively promotes low-rank structure, leading to improved estimation accuracy in high-dimensional settings.