[Paper Review] How close is the sample covariance matrix to the actual covariance matrix?

Roman Vershynin|arXiv (Cornell University)|Apr 20, 2010

Random Matrices and Applications22 references4 citations

TL;DR

This paper investigates the sample size N required to approximate the true covariance matrix of a high-dimensional random vector in the operator norm, proving that N = O(n) suffices up to a logarithmic factor for distributions with finite fourth moments. It establishes that the sample covariance matrix converges to the true covariance matrix at a rate of (n/N)^{1/2 - 2/q} with high probability, improving upon prior results for sub-exponential and sub-gaussian distributions.

ABSTRACT

Given a probability distribution in R^n with general (non-white) covariance, a classical estimator of the covariance matrix is the sample covariance matrix obtained from a sample of N independent points. What is the optimal sample size N = N(n) that guarantees estimation with a fixed accuracy in the operator norm? Suppose the distribution is supported in a centered Euclidean ball of radius \sqrt{n}. We conjecture that the optimal sample size is N = O(n) for all distributions with finite fourth moment, and we prove this up to an iterated logarithmic factor. This problem is motivated by the optimal theorem of Rudelson which states that N = O(n \log n) for distributions with finite second moment, and a recent result of Adamczak, Litvak, Pajor and Tomczak-Jaegermann which guarantees that N = O(n) for sub-exponential distributions.

Motivation & Objective

• To determine the minimal sample size N(n, ε) that ensures high-probability approximation of the true covariance matrix by the sample covariance matrix within error ε in the operator norm.
• To close the gap between known results for sub-exponential distributions (N=O(n)) and general distributions with finite fourth moments, conjecturing that N=O(n) is optimal.
• To extend the analysis beyond sub-gaussian and sub-exponential distributions to general distributions with only finite fourth moments.
• To provide a refined concentration bound for the operator norm of the difference between sample and true covariance matrices using truncation and moment-based techniques.

Proposed method

• Uses a truncation argument to decompose the estimation error into contributions from small and large coefficients of the random vectors.
• Applies moment assumptions (2.2) with finite q > 4 to control tail behavior and derive probabilistic bounds on large coefficients.
• Employs Theorem 5.1 on the concentration of the sum of rank-one projections to bound the contribution of large coefficients in the sample covariance matrix.
• Applies Hölder’s and Markov’s inequalities to control the expectation of squared coefficients beyond a threshold B.
• Chooses a threshold B = (N/n)^{2/q} to balance the trade-off between bias and variance terms in the error decomposition.
• Combines bounds on three components: the main term (I₁), the large coefficient sum (I₂), and the expected large coefficient contribution (I₃), to derive the final rate.

Experimental results

Research questions

• RQ1 What is the optimal sample size N(n) required to approximate the true covariance matrix within a fixed error ε in the operator norm for general distributions with finite fourth moments?
• RQ2 Can the sample size N=O(n) be achieved for distributions with only finite fourth moments, as conjectured, or is a larger N required?
• RQ3 How does the convergence rate of the sample covariance matrix depend on the moment structure of the underlying distribution?
• RQ4 To what extent can truncation and moment-based concentration techniques improve operator norm bounds beyond sub-gaussian or sub-exponential assumptions?

Key findings

• For distributions with finite q > 4 moments, the sample covariance matrix approximates the true covariance matrix with high probability at a rate of (n/N)^{1/2 - 2/q} in the operator norm.
• The required sample size is N = O(n) up to a factor of (log log n)^2, improving upon the O(n log n) bound for general second-moment distributions.
• The bound holds under the assumption that the distribution is supported in a Euclidean ball of radius O(√n), which is a natural condition for high-dimensional concentration.
• The analysis shows that the operator norm error is dominated by the interplay between the dimension n, sample size N, and the moment order q of the distribution.
• The truncation-based method successfully controls the contribution of heavy-tailed coefficients, enabling the extension of results from sub-exponential to general finite-moment distributions.