[Paper Review] On Concentration Inequalities for Random Matrix Products

Tarun Kathuria, Satyaki Mukherjee+1 more|arXiv (Cornell University)|Mar 13, 2020

Random Matrices and Applications4 references4 citations

TL;DR

This paper presents a sharp concentration inequality for normalized random matrix products of the form $\prod_{i=1}^n \left(I + \frac{X_i}{n}\right)$, where $X_i$ are i.i.d. $d \times d$ random matrices with mean $\mu$ and operator norm bounded by $L$. Using a matrix Doob martingale and the Matrix Freedman inequality, the authors achieve optimal $n$ and $d$ dependence up to constant factors, with a tail bound of $\mathsf{Pr}\left[\left\|f(X_1,\ldots,X_n) - e^{\mu}\right\|_{\mathsf{op}} \geq t\right] \leq 2d \cdot \exp\left(-c n t^2 / (L^2 e^{2L})\right)$, matching the matrix Bernstein inequality up to constants.

ABSTRACT

Consider $n$ complex random matrices $X_1,\ldots,X_n$ of size $d imes d$ sampled i.i.d. from a distribution with mean $E[X]=μ$. While the concentration of averages of these matrices is well-studied, the concentration of other functions of such matrices is less clear. One function which arises in the context of stochastic iterative algorithms, like Oja's algorithm for Principal Component Analysis, is the normalized matrix product defined as $\prod\limits_{i=1}^{n}\left(I + \frac{X_i}{n} ight).$ Concentration properties of this normalized matrix product were recently studied by \cite{HW19}. However, their result is suboptimal in terms of the dependence on the dimension of the matrices as well as the number of samples. In this paper, we present a stronger concentration result for such matrix products which is optimal in $n$ and $d$ up to constant factors. Our proof is based on considering a matrix Doob martingale, controlling the quadratic variation of that martingale, and applying the Matrix Freedman inequality of Tropp \cite{TroppIntro15}.

Motivation & Objective

• To address the lack of tight concentration bounds for normalized matrix products in stochastic iterative algorithms such as Oja’s algorithm for PCA.
• To overcome the suboptimal $\log^2 n$ factor in prior work by Henriksen and Ward (2020), which used partitioning and matrix Bernstein on grouped terms.
• To establish a concentration result for $\prod_{i=1}^n \left(I + \frac{X_i}{n}\right)$ that matches the matrix Bernstein inequality for sums in terms of $n$ and $d$ dependence, up to constant factors.
• To demonstrate that the $L^2 e^{2L}$ dependence in the variance proxy is necessary, even in the scalar case, by constructing a lower bound example.

Proposed method

• Construct a matrix Doob martingale $Y_k = \mathbb{E}[f(X_1,\ldots,X_n) \mid X_1,\ldots,X_k] - \mathbb{E}[f(X_1,\ldots,X_n) \mid X_1,\ldots,X_{k-1}]$ to represent the incremental change in the matrix product upon revealing each $X_i$.
• Bound the spectral norm of each martingale increment $Y_k$ by $\frac{2Le^L}{n}$ using submultiplicativity and the bound $\|X_i\|_{\mathsf{op}} \leq L$.
• Control the predictable quadratic variation $\mathbb{E}[Y_k Y_k^* \mid X_1,\ldots,X_{k-1}]$ by showing its norm is at most $\frac{4L^2 e^{2L}}{n^2}$, leading to a total variation bound of $\frac{4L^2 e^{2L}}{n}$.
• Apply the Matrix Freedman inequality (Tropp, 2015) to the martingale, which provides a tail bound depending on the maximum increment norm and the total predictable variation.
• Derive the final concentration inequality $\mathsf{Pr}\left[\|f(X_1,\ldots,X_n) - e^\mu\|_{\mathsf{op}} \geq t\right] \leq 2d \cdot \exp\left(-c n t^2 / (L^2 e^{2L})\right)$ under the condition $t \leq Le^L \sqrt{\log d / n}$.

Experimental results

Research questions

• RQ1 Can the concentration of normalized matrix products $\prod_{i=1}^n \left(I + \frac{X_i}{n}\right)$ be bounded with optimal dependence on $n$ and $d$, matching the matrix Bernstein inequality for sums?
• RQ2 Is the $\log^2 n$ factor loss in the prior result by Henriksen and Ward (2020) necessary, or can it be avoided with a different proof technique?
• RQ3 What is the correct dependence on the matrix norm bound $L$ in the concentration tail, and is $L^2 e^{2L}$ necessary rather than $L^2$?
• RQ4 Can a martingale-based approach using the Matrix Freedman inequality yield tighter bounds than partitioning-based methods?
• RQ5 Is the $e^{O(L)}$ factor in the variance proxy unavoidable, even in the scalar case?

Key findings

• The paper establishes a concentration inequality for the normalized matrix product $f(X_1,\ldots,X_n) = \prod_{i=1}^n \left(I + \frac{X_i}{n}\right)$ with tail bound $\mathsf{Pr}\left[\|f(X_1,\ldots,X_n) - e^\mu\|_{\mathsf{op}} \geq t\right] \leq 2d \cdot \exp\left(-c n t^2 / (L^2 e^{2L})\right)$, which is optimal in $n$ and $d$ up to constant factors.
• The bound is derived via a matrix Doob martingale and the Matrix Freedman inequality, avoiding the $\log^2 n$ factor loss present in the prior work by Henriksen and Ward (2020).
• The spectral norm of each martingale increment is bounded by $\frac{2Le^L}{n}$, and the total predictable quadratic variation is bounded by $\frac{4L^2 e^{2L}}{n}$, both crucial for applying Matrix Freedman.
• A lower bound example with i.i.d. $X_i$ taking values $0$ and $2L$ with equal probability shows that the $L^2 e^{2L}$ dependence in the variance proxy is necessary, even for scalar products.
• The result matches the matrix Bernstein inequality for sums in terms of $n$ and $d$ dependence, with only a constant factor difference due to the $e^{2L}$ factor in the variance proxy.
• The authors confirm that the $e^{O(L)}$ dependence cannot be removed if the bound is expressed solely in terms of $L$, as demonstrated by the scalar counterexample.