[Paper Review] Squared-Norm Empirical Process in Banach Space

Vincent Q. Vu, Jing Lei|arXiv (Cornell University)|Dec 4, 2013

Point processes and geometric inequalities2 references3 citations

TL;DR

This paper extends Mendelson's result on the supremum of a quadratic empirical process to squared norms of functions in a Banach space, using symmetrization and subadditivity of the generic chaining functional. The key contribution is a high-probability bound for the supremum deviation of empirical squared norms, with applications to linear processes in sample covariance matrices indexed by finite-rank positive definite matrices.

ABSTRACT

This note extends a recent result of Mendelson on the supremum of a quadratic process to squared norms of functions taking values in a Banach space. Our method of proof is a reduction by a symmetrization argument and observation about the subadditivity of the generic chaining functional. We provide an application to the supremum of a linear process in the sample covariance matrix indexed by finite rank, positive definite matrices.

Motivation & Objective

• To generalize Mendelson's supremum bound for quadratic processes from real-valued functions to functions taking values in a Banach space.
• To establish a high-probability tail bound for the empirical squared norm process: $\left|\frac{1}{n}\sum_{i=1}^{n}\|g(X_i)\|^2 - \mathbb{E}\|g(X_i)\|^2\right|$.
• To apply the derived bound to the supremum of linear processes in the sample covariance matrix indexed by finite-rank, positive definite matrices.
• To quantify the complexity of the function class $G$ via Orlicz norms and generic chaining functional $\gamma_2$ under a suitable metric.
• To provide a framework for analyzing empirical processes involving operator norms and random matrix statistics in high-dimensional settings.

Proposed method

• Use a symmetrization argument to reduce the Banach space-valued squared norm process to a form amenable to existing quadratic process bounds.
• Introduce a metric $d(g_1, g_2) = \|\|g_1(X_1) - g_2(X_1)\|\|_{\psi_2}$ on the function class $G$ to measure complexity.
• Apply the subadditivity property of the generic chaining functional $\gamma_2$ to control the complexity of the union of symmetric classes.
• Relate the $\psi_2$-norm of $\|A^T X_1\|$ to the Frobenius norm of $A$ via the inequality $\|\|A^T Z\|\|_{\psi_2} \leq \|A\|_F \|Z\|_{\psi_2}$.
• Use the majorizing measure theorem to bound $\gamma_2(G, d)$ in terms of the expected supremum of a Gaussian process $\mathbb{E} \sup_{A \in \mathcal{A}} \langle \mathcal{Z}, A \rangle$.
• Derive a tail bound for the empirical process deviation using the derived complexity parameters and apply it to the sample covariance matrix.

Experimental results

Research questions

• RQ1 Can the supremum deviation of the empirical squared norm process in a Banach space be bounded with exponential tail probability?
• RQ2 How does the complexity of the function class $G$—measured via $\gamma_2$ and Orlicz norms—control the deviation of the empirical squared norm?
• RQ3 What is the connection between the generic chaining functional $\gamma_2$ and the subadditivity of the metric entropy in the context of symmetrized processes?
• RQ4 How can the bound on the empirical squared norm process be applied to linear processes in the sample covariance matrix?
• RQ5 What is the role of the Frobenius norm of $A$ and the $\psi_2$-norm of the random vector $X_1$ in determining the rate of convergence?

Key findings

• The paper establishes a high-probability bound for the supremum of the empirical squared norm process: $\left|\frac{1}{n}\sum_{i=1}^{n}\|g(X_i)\|^2 - \mathbb{E}\|g(X_i)\|^2\right| \leq c_3 t \left\{ \frac{d_{\psi_1}(G) \gamma_2(G,d)}{\sqrt{n}} + \frac{\gamma_2^2(G,d)}{n} \right\}$ with probability at least $1 - 2\exp(-c_2 t^{2/5})$.
• For linear processes with $g(x) = A^T x$, the bound is expressed in terms of $\sigma = \sup_{\|u\|_2=1} \|\langle X_1, u \rangle\|_{\psi_2}$, $\sup_{A \in \mathcal{A}} \|A\|_F$, and $\mathbb{E} \sup_{A \in \mathcal{A}} \langle \mathcal{Z}, A \rangle$.
• The bound implies that $\mathbb{E} \sup_{A \in \mathcal{A}} |\langle S_n - \Sigma, AA^T \rangle| \leq c \left\{ \frac{\sigma^2 \sup_A \|A\|_F}{\sqrt{n}} \mathbb{E} \sup_A \langle \mathcal{Z}, A \rangle + \frac{\sigma^2}{n} \left( \mathbb{E} \sup_A \langle \mathcal{Z}, A \rangle \right)^2 \right\}$.
• The result holds under symmetry of the class $\mathcal{A}$ and assumes $X_1$ has mean zero and sub-Gaussian-like tails via the $\psi_2$-norm.
• The application to sample covariance matrices shows that the deviation of the empirical covariance from the true covariance is controlled by the complexity of the matrix class $\mathcal{A}$.
• The derived bound is sharp in the sense that it matches the known rates for Gaussian processes and extends them to non-Gaussian, heavy-tailed settings via Orlicz norms.