[Paper Review] Relative concentration bounds for the spectrum of kernel matrices

Ernesto Araya Valdivia|arXiv (Cornell University)|Dec 5, 2018

Random Matrices and Applications44 references4 citations

TL;DR

This paper establishes relative concentration inequalities for individual eigenvalues of kernel matrices, scaling with the eigenvalue itself to achieve faster-than-parametric, often exponential, convergence rates. The approach applies to indefinite (non-positive) kernels—critical for network analysis—using Sobolev-type regularity and spectral expansion convergence, yielding tighter bounds than classical Weyl-type inequalities.

ABSTRACT

In this paper we study the concentration properties for the eigenvalues of kernel matrices, which are central objects in a wide range of kernel methods and, more recently, in network analysis. We present a set of concentration inequalities tailored for each individual eigenvalue of the kernel matrix with respect to its known asymptotic limit. The inequalities presented here are of relative type, meaning that they scale with the eigenvalue in consideration, which results in convergence rates that vary across the spectrum. The rates we obtain here are faster than the typical $Ø(\frac{1}{\sqrt n})$ and are often exponential, depending on regularity assumptions of Sobolev type. One key feature of our results is that they apply to non positive kernels, which is fundamental in the context of network analysis. We show how our results are well suited for the study of dot product kernels, which are related to random geometric graphs on the sphere, via the graphon formalism. We illustrate our results by applying them to a variety of dot product kernels on the sphere and to the one dimensional Gaussian kernel.

Motivation & Objective

• To develop tighter, relative concentration inequalities for individual eigenvalues of kernel matrices, improving upon absolute bounds like those in Weyl's inequality.
• To extend concentration results to indefinite (non-positive) kernels, which are essential in modern network analysis via the graphon formalism.
• To achieve convergence rates faster than the standard $\mathcal{O}(n^{-1/2})$ parametric rate, particularly exponential or near-exponential rates under Sobolev-type regularity.
• To unify theoretical analysis of dot product kernels on the sphere and one-dimensional Gaussian kernels under a common framework of spectral expansion convergence.
• To provide a foundation for extending Hilbert space and RKHS-based methods to indefinite kernels, which currently require positive semidefiniteness.

Proposed method

• Derive relative concentration inequalities of Weyl-type, where the deviation of each eigenvalue scales with its own magnitude: $|\lambda_i(T_W) - \lambda_i(T_n)| \lesssim |\lambda_i| n^{-q}$ for $0 < q < 1$.
• Use spectral expansion of the kernel $W$ and impose regularity conditions on eigenvalues (polynomial or exponential decay) and eigenvectors (uniform boundedness).
• Apply a three-step framework: (1) spectral approximation via truncation of the integral operator, (2) perturbation analysis of the finite-rank approximation, and (3) concentration of the residual matrix $E_R$.
• Employ matrix concentration tools on the residual operator norm $\|E_R\|_{\text{op}}$, using bounds derived from uniform control of eigenfunctions and decay assumptions.
• Introduce a partitioning strategy for the tail indices $[R, \infty) \cap \mathbb{N}$ to refine concentration bounds when the operator norm estimate is loose.
• Leverage the connection between dot product kernels on the sphere and graphons, particularly the Erdős–Rényi graphon, to validate results on real-world network models.

Experimental results

Research questions

• RQ1 Can relative concentration inequalities for kernel matrix eigenvalues achieve faster-than-parametric convergence rates in the low-dimensional setting?
• RQ2 How can such bounds be extended to indefinite kernels, which are common in network analysis and graphon modeling?
• RQ3 What regularity assumptions on eigenvalues and eigenfunctions are sufficient to ensure tight concentration with exponential rates?
• RQ4 Can the spectral expansion convergence be controlled uniformly to enable relative error bounds without requiring positive semidefiniteness?
• RQ5 What improvements are possible in the concentration step when the residual matrix norm is overestimated, especially for kernels with slow eigenvalue decay?

Key findings

• The paper establishes relative concentration bounds of the form $|\lambda_i(T_W) - \lambda_i(T_n)| \lesssim_{\alpha} e^{-1.6i} n^{-1/2}$ with probability greater than $1 - \alpha$, for dot product kernels on the sphere and the 1D Gaussian kernel.
• For kernels satisfying $\text{H}_2$ with $s=0$ and $\delta = \log 5$, the convergence rate is $O(e^{-1.6i} n^{-1/2})$, demonstrating exponential decay in the eigenvalue index $i$.
• The results apply to indefinite kernels, such as the logistic and proximity graphons, which are not covered by existing positive-semidefinite-based methods.
• The method achieves faster convergence than standard $\mathcal{O}(n^{-1/2})$ rates, with exponential or near-exponential decay depending on eigenvalue decay and eigenvector regularity.
• The approach is robust to the lack of pointwise equality in spectral expansions by using uniform bounds on eigenfunctions and truncation-based approximation.
• The framework can be extended to $m$-fold compositions of the kernel operator, though this may degrade convergence rates, indicating a trade-off between generality and tightness.