[论文解读] The Sample Complexity of Dictionary Learning

Daniel Vainsencher, Shie Mannor+1 more|Nov 24, 2010

Sparse and Compressive Sensing Techniques参考文献 14被引用 4

一句话总结

该论文在系数选择的两种约束条件下——$l_1$-正则化与$k$-稀疏表示——建立了字典学习的泛化界。在低Babel函数假设下（该假设在高维中以高概率成立），通过局部Rademacher复杂度，分别得到了$O\left(\sqrt{np\log(m\lambda)/m}\right)$与$O\left(\sqrt{np\log(mk)/m}\right)$的样本复杂度界，且收敛速度达到快速的$1/m$阶。

ABSTRACT

A large set of signals can sometimes be described sparsely using a dictionary, that is, every element can be represented as a linear combination of few elements from the dictionary. Algorithms for various signal processing applications, including classification, denoising and signal separation, learn a dictionary from a set of signals to be represented. Can we expect that the representation found by such a dictionary for a previously unseen example from the same source will have L_2 error of the same magnitude as those for the given examples? We assume signals are generated from a fixed distribution, and study this questions from a statistical learning theory perspective. We develop generalization bounds on the quality of the learned dictionary for two types of constraints on the coefficient selection, as measured by the expected L_2 error in representation when the dictionary is used. For the case of l_1 regularized coefficient selection we provide a generalization bound of the order of O(sqrt(np log(m lambda)/m)), where n is the dimension, p is the number of elements in the dictionary, lambda is a bound on the l_1 norm of the coefficient vector and m is the number of samples, which complements existing results. For the case of representing a new signal as a combination of at most k dictionary elements, we provide a bound of the order O(sqrt(np log(m k)/m)) under an assumption on the level of orthogonality of the dictionary (low Babel function). We further show that this assumption holds for most dictionaries in high dimensions in a strong probabilistic sense. Our results further yield fast rates of order 1/m as opposed to 1/sqrt(m) using localized Rademacher complexity. We provide similar results in a general setting using kernels with weak smoothness requirements.

研究动机与目标

• 建立从有限样本中学习字典的统计泛化界。
• 分析使学习到的字典能泛化到未见信号所需的样本复杂度。
• 量化字典大小、稀疏度以及几何性质（如Babel函数）对泛化误差的影响。
• 将结果扩展至具有弱光滑性假设的核基字典学习。
• 利用局部Rademacher复杂度实现快速收敛速率，优于标准的$1/\sqrt{m}$速率。

提出的方法

• 使用统计学习理论，推导所有可接受字典上的统一收敛界。
• 应用局部Rademacher复杂度，实现$1/m$阶的快速收敛速率，而非$1/\sqrt{m}$。
• 引入Babel函数作为字典相干性的关键度量，并假设其值较小以保证泛化性能。
• 推导由核映射诱导的函数类的覆盖数界，从而实现在再生核Hilbert空间中的泛化。
• 利用特征映射与核函数的Hölder连续性，控制度量熵与覆盖大小。
• 建立两种系数约束下的边界：$l_1$-范数有界（$R_\lambda$）与$k$-稀疏（$H_k$）表示。

实验结果

研究问题

• RQ1 从$m$个样本中学习到的字典，为使在未见信号上具有低期望$L_2$误差，所需的样本复杂度是多少？
• RQ2 $l_1$-正则化约束如何影响泛化误差？其对$l_1$-范数界$\lambda$的依赖关系如何？
• RQ3 $k$-稀疏表示约束如何影响泛化性能？Babel函数在此中起什么作用？
• RQ4 能否将泛化界扩展至具有弱光滑性假设的核基字典学习？
• RQ5 在何种条件下，字典学习中会出现$1/m$阶的快速收敛速率？

主要发现

• 对于$l_1$-正则化系数选择，泛化误差被界为$O\left(\sqrt{np\log(m\lambda)/m}\right)$，显示出对$\lambda$的对数依赖。
• 对于$k$-稀疏表示，边界为$O\left(\sqrt{np\log(mk)/m}\right)$，前提是Babel函数值较小。
• 在高维中，大多数字典的Babel函数值较小这一假设在经验上成立，具有高概率。
• 通过局部Rademacher复杂度，实现了$1/m$阶的快速收敛速率，优于标准的$1/\sqrt{m}$速率。
• 结果可扩展至使用Hölder连续特征映射的核基设置，推导出核诱导函数类的覆盖数界。
• 边界表明，字典学习算法应通过最小化Babel函数来正则化，以获得更好的泛化性能。