[论文解读] Approximation beats concentration? An approximation view on inference with smooth radial kernels

Mikhail A. Belkin|arXiv (Cornell University)|Jan 10, 2018

Gaussian Processes and Bayesian Inference被引用 31

一句话总结

本文运用逼近理论分析核方法中的光滑径向核，表明特征值近乎指数衰减，且RKHS中的函数在特征基下系数迅速衰减，从而呈现出有效的低秩结构。其核心贡献在于‘逼近优于集中’现象：与标准集中不等式结果相比，逼近理论给出的特征值衰减和肥沙射维数的界更紧致，且与测度无关。

ABSTRACT

Positive definite kernels and their associated Reproducing Kernel Hilbert Spaces provide a mathematically compelling and practically competitive framework for learning from data. In this paper we take the approximation theory point of view to explore various aspects of smooth kernels related to their inferential properties. We analyze eigenvalue decay of kernels operators and matrices, properties of eigenfunctions/eigenvectors and "Fourier" coefficients of functions in the kernel space restricted to a discrete set of data points. We also investigate the fitting capacity of kernels, giving explicit bounds on the fat shattering dimension of the balls in Reproducing Kernel Hilbert spaces. Interestingly, the same properties that make kernels very effective approximators for functions in their "native" kernel space, also limit their capacity to represent arbitrary functions. We discuss various implications, including those for gradient descent type methods. It is important to note that most of our bounds are measure independent. Moreover, at least in moderate dimension, the bounds for eigenvalues are much tighter than the bounds which can be obtained from the usual matrix concentration results. For example, we see that the eigenvalues of kernel matrices show nearly exponential decay with constants depending only on the kernel and the domain. We call this "approximation beats concentration" phenomenon as even when the data are sampled from a probability distribution, some of their aspects are better understood in terms of approximation theory.

研究动机与目标

• 用逼近理论重新框架化核方法的分析，而非依赖集中不等式。
• 理解光滑径向核在拟合任意函数时的推断限制。
• 为RKHS球的特征值衰减和肥沙射维数提供与测度无关的界。
• 阐明核带宽与RKHS容量之间的关系，表明更宽的核产生更小的函数空间。
• 解释为何使用光滑核的梯度下降在拟合随机标签时表现困难，这与特征基下的系数衰减及计算复杂度相关。

提出的方法

• 运用逼近理论分析核算子和矩阵的特征值衰减，表明其衰减近乎指数，且与数据测度无关。
• 利用RKHS的傅里叶域表征，比较不同核带宽所诱导的函数空间。
• 通过系数衰减和谱性质，推导RKHS球的肥沙射维数界。
• 证明核矩阵的前导特征向量可被数据点处核函数的线性组合近乎指数逼近。
• 应用特征函数逼近结果，表明前导特征向量张成的空间对测度变化具有鲁棒性。
• 利用光滑径向核的傅里叶变换快速衰减的性质，推导RKHS范数和函数空间包含关系的界。

实验结果

研究问题

• RQ1 为何标准矩阵集中结果无法捕捉核矩阵特征值的真实衰减速率？
• RQ2 即使在大数据下，光滑径向核在多大程度上限制了核方法的拟合能力？
• RQ3 核带宽的选择如何影响相应RKHS的函数空间？
• RQ4 逼近理论能否为RKHS球的肥沙射维数提供比基于集中方法更紧致的界？
• RQ5 为何光滑核难以拟合随机标签，这与特征基下系数衰减有何关联？

主要发现

• 光滑径向核矩阵的特征值衰减近乎指数，衰减常数仅依赖于核函数和维度，与数据测度无关。
• 光滑核RKHS中的函数在特征基下的傅里叶系数衰减近乎指数，与数据测度无关。
• 核矩阵前导特征向量的张成空间可被数据点处核函数的线性组合近乎指数逼近。
• RKHS中半径为R的球的肥沙射维数在R/γ下为多对数级，限制了正则化和基于梯度方法的拟合能力。
• 更宽的高斯核产生更小的RKHS空间：更宽核的RKHS包含于更窄核的RKHS中，且范数按σ^{-d/2}缩放。
• 使用光滑核的梯度下降在拟合随机标签时面临超立方复杂度，原因在于特征基中需使用指数级小的系数。