[论文解读] Recklessly approximate sparse coding

Misha Denil|arXiv (Cornell University)|Jan 1, 2012

Sparse and Compressive Sensing Techniques参考文献 40被引用 12

一句话总结

本文证明，软阈值特征——作为高效图像分类的稀疏编码替代方法——在数学上等价于非负稀疏编码目标函数上的近端梯度下降的单步迭代。这为它们表现出色的性能提供了首个理论依据，表明即使使用近似稀疏编码解，也能生成有效的分类器。

ABSTRACT

Introduction of the so called “K-means” or “triangle” features in Coates, Lee and Ng, 2011 caused significant discussion in the deep learning community. These simple features are able to achieve state of the art performance on standard image classification benchmarks, outperforming much more sophisticated methods including deep belief networks, convolutional nets, factored RBMs, mcRBMs, convolutional RBMs, sparse autoencoders and several others. Moreover, these features are extremely simple and easy to compute. Several intuitive arguments have been put forward to describe this remarkable performance, yet no mathematical justification has been offered. In Coates and Ng, 2011, the authors improve on the triangle features with “soft threshold” features, adding a hyperparameter to tune performance, and compare these features to sparse coding. Both soft thresholding and sparse coding are found to often yield similar classification results, though soft threshold features are much faster to compute. The main result of this thesis is to show that the soft threshold features are realized as a single step of proximal gradient descent on a non-negative sparse coding objective. This result is important because it provides an explanation for the success of the soft threshold features and shows that even very approximate solutions to the sparse coding problem are sufficient to build effective classifiers.

研究动机与目标

• 为软阈值特征在图像分类中表现优异提供数学解释。
• 研究软阈值特征与稀疏编码在特征学习背景下的关系。
• 证明稀疏编码的近似解仍可产生高度有效的分类器。
• 形式化软阈值与非负稀疏编码目标函数上优化之间的联系。

提出的方法

• 提出一个非负稀疏编码目标函数，用于建模软阈值特征的学习过程。
• 推导出软阈值特征计算为非负稀疏编码目标函数上的单步近端梯度下降。
• 使用近端梯度下降迭代最小化带稀疏性与非负性约束的目标函数。
• 在标准图像分类基准上比较软阈值特征与完整稀疏编码的性能。
• 在软阈值中引入一个超参数，以调节特征稀疏性与性能。
• 在标准数据集上进行实证评估，以验证理论等价性与性能表现。

实验结果

研究问题

• RQ1 为何软阈值特征尽管结构简单，却能在图像分类基准上实现最先进性能？
• RQ2 从优化视角看，软阈值特征与稀疏编码有何关联？
• RQ3 在非负稀疏编码目标函数上执行单步近端梯度下降，能否重现完整稀疏编码的性能？
• RQ4 稀疏编码问题的近似解在多大程度上仍能生成有效的分类器？

主要发现

• 软阈值特征在数学上等价于非负稀疏编码目标函数上的单步近端梯度下降。
• 所提方法为软阈值特征在实践中取得成功提供了首个理论解释。
• 软阈值特征在分类性能上可与完整稀疏编码相媲美，同时计算速度显著更快。
• 即使仅对稀疏编码目标函数执行单步优化，也能生成对分类极为有效的特征。
• 结果证实，稀疏编码的近似解已足够用于构建高性能图像分类器。