[Paper Review] Fast Inference in Sparse Coding Algorithms with Applications to Object Recognition

Koray Kavukcuoglu, Marc’Aurelio Ranzato+1 more|arXiv (Cornell University)|Oct 18, 2010

Sparse and Compressive Sensing TechniquesEngineering18 references196 citations

TL;DR

This paper proposes Predictive Sparse Decomposition (PSD), a fast, feed-forward neural network-based regressor that approximates optimal sparse codes from sparse coding algorithms. By jointly training the basis functions and the predictor, PSD achieves over 100x speedup over exact optimization methods while matching or exceeding recognition accuracy on object recognition tasks, enabling real-time inference with stable, smooth representations.

ABSTRACT

Adaptive sparse coding methods learn a possibly overcomplete set of basis functions, such that natural image patches can be reconstructed by linearly combining a small subset of these bases. The applicability of these methods to visual object recognition tasks has been limited because of the prohibitive cost of the optimization algorithms required to compute the sparse representation. In this work we propose a simple and efficient algorithm to learn basis functions. After training, this model also provides a fast and smooth approximator to the optimal representation, achieving even better accuracy than exact sparse coding algorithms on visual object recognition tasks.

Motivation & Objective

• To address the high computational cost of inference in sparse coding algorithms, which limits their use in real-time vision systems.
• To develop a fast, smooth, and accurate approximator for optimal sparse representations without sacrificing recognition performance.
• To jointly train basis functions and a predictor so that the resulting representations are both sparse and easily predictable.
• To improve stability of sparse representations across time-varying inputs, such as video frames, compared to exact optimization methods.
• To enable efficient, convolutional application of sparse coding in hierarchical vision systems for object recognition.

Proposed method

• Propose a non-linear feed-forward regressor F(Y; G, W, D) = G·tanh(WY + D) that maps image patches Y to sparse representations Z.
• Train the regressor jointly with the basis matrix B using a compound loss function: ||Y - BZ||² + λ||Z||₁ + α||Z - F(Y; P_f)||².
• Use the hyperbolic tangent non-linearity to enable sparse output despite overcomplete, non-orthogonal bases.
• Introduce a diagonal gain matrix G to compensate for unit-norm basis scaling, ensuring consistent reconstruction.
• Optimize all parameters (B, G, W, D) end-to-end to produce basis functions and a predictor that jointly minimize reconstruction error, sparsity, and prediction error.
• Apply the trained predictor in a convolutional feature extraction pipeline with ReLU-like rectification and average pooling before SVM classification.

Experimental results

Research questions

• RQ1 Can a feed-forward neural network be trained to predict optimal sparse codes from sparse coding algorithms with high accuracy and low computational cost?
• RQ2 Does the proposed joint training of basis functions and predictor lead to representations that are both sparse and easily predictable?
• RQ3 Can the predicted representations achieve higher recognition accuracy than exact sparse coding algorithms, despite being approximations?
• RQ4 How does the stability of the predicted representations compare to those from exact optimization methods on natural video sequences?
• RQ5 What is the trade-off between inference speed, sparsity, and recognition accuracy in real-world object recognition tasks?

Key findings

• The PSD predictor achieves over 100x speedup in feature extraction compared to the fastest exact sparse coding algorithm (feature sign), with a speedup exceeding 800x at optimal sparsity levels.
• Despite being an approximation, the PSD predictor achieves higher object recognition accuracy on the Caltech-101 dataset than exact sparse coding algorithms, even when the latter are optimized for sparsity.
• The predicted representations are more stable over time, with significantly lower sign transition rates between consecutive video frames than those from exact optimization, indicating smoother dynamics.
• The joint training of basis functions and predictor results in a model that generalizes better, as evidenced by superior performance with only 30 training images per class (53% accuracy).
• The method reduces redundancy in convolutional feature maps by learning localized, oriented edge-like basis functions that are efficiently shared across spatial locations.
• The compound loss function enables the model to learn basis functions and predictors that are both sparse and highly predictable, leading to efficient and accurate inference.