[Paper Review] A Finite-Sample Deviation Bound for Stable Autoregressive Processes

Rodrigo A. González, Cristian R. Rojas|arXiv (Cornell University)|Dec 17, 2019

Statistical Methods and Inference19 references4 citations

TL;DR

This paper derives a finite-sample deviation bound for stable autoregressive processes of order $ n $, using matrix norms and concentration inequalities to quantify the deviation of the sample covariance from its population counterpart. The key contribution is a high-probability bound on estimation error that scales favorably with sample size $ N $, system dimension $ n $, and noise variance $ \sigma^2 $, with explicit dependence on model stability and design matrix structure.

ABSTRACT

In this paper, we study non-asymptotic deviation bounds of the least squares estimator in Gaussian AR($n$) processes. By relying on martingale concentration inequalities and a tail-bound for $χ^2$ distributed variables, we provide a concentration bound for the sample covariance matrix of the process output. With this, we present a problem-dependent finite-time bound on the deviation probability of any fixed linear combination of the estimated parameters of the AR$(n)$ process. We discuss extensions and limitations of our approach.

Motivation & Objective

• To establish a non-asymptotic probabilistic bound on the deviation of the sample covariance matrix from its population counterpart in stable autoregressive processes.
• To quantify the estimation error in finite samples for AR(n) models under general noise and stability conditions.
• To derive a high-probability upper bound on the deviation of the empirical covariance from the true covariance, incorporating model parameters and sample size.
• To analyze the impact of model stability, noise variance, and sample size on the finite-sample performance of autoregressive estimators.
• To provide a rigorous, non-asymptotic deviation bound that accounts for the structure of the design matrix and the spectral properties of the autoregressive coefficient matrix.

Proposed method

• Define the long-run covariance matrix $ \overline{V} = \sigma^2 \sum_{i=0}^\infty A^i B B^\top (A^\top)^i $, representing the asymptotic covariance of the process.
• Construct data-dependent bounds $ V_{\text{dn}} $ and $ V_{\text{up}} $ using perturbations of $ \overline{V} $ scaled by $ \epsilon \sigma^2 \sum_{i=0}^\infty A^i (A^\top)^i $, capturing lower and upper deviation envelopes.
• Introduce a deviation probability bound $ \delta(\epsilon, N) $ combining four exponential terms derived from concentration inequalities for sub-Gaussian and sub-exponential processes.
• Use matrix projection via $ \begin{bmatrix} I_n & 0 \end{bmatrix} $ to extract the relevant covariance submatrix from the full state-space covariance.
• Apply matrix norm and spectral radius arguments to control the growth of the deviation terms, ensuring stability under model assumptions.
• Combine tail bounds from sub-Gaussian and sub-exponential concentration to derive a composite bound on the probability of large deviations.

Experimental results

Research questions

• RQ1 How does the estimation error of the sample covariance matrix in an AR(n) process behave in finite samples?
• RQ2 What is the rate at which the empirical covariance converges to the true covariance under model stability and sub-Gaussian noise?
• RQ3 How do the model parameters, including the coefficient matrix $ A $, noise variance $ \sigma^2 $, and sample size $ N $, affect the deviation probability?
• RQ4 Can a high-probability bound on the deviation be derived that is both non-asymptotic and sensitive to the system's spectral properties?
• RQ5 What is the role of the perturbation parameter $ \epsilon $ in controlling the confidence level of the deviation bound?

Key findings

• The deviation probability $ \delta(\epsilon, N) $ decays exponentially in $ N - n $, indicating strong concentration of the sample covariance around the true value for large samples.
• The bound $ \delta(\epsilon, N) $ includes four terms: one from sub-Gaussian concentration, one from sub-exponential tail behavior, one from the $ \ell^2 $-norm of the coefficient vector $ \theta $, and one from a $ \sqrt{N} $-scaled exponential decay.
• The upper and lower bounds $ V_{\text{up}} $ and $ V_{\text{dn}} $ are constructed to capture the range of possible deviations of the empirical covariance from the true covariance, scaled by $ (N - n) $.
• The bound is valid under the assumption that the AR process is stable, ensuring the convergence of the infinite series in $ \overline{V} $.
• The deviation bound is non-asymptotic and explicitly depends on $ \mathbb{E}\{y_1^2\} $, $ \|\theta\|_2 $, and the noise variance $ \sigma^2 $, reflecting model and data characteristics.
• The result provides a finite-sample confidence region for the covariance matrix of the AR(n) process, with explicit dependence on model stability and sample size.