[Paper Review] Deep Network Classification by Scattering and Homotopy Dictionary Learning

J. Zarka, Louis Thiry+2 more|arXiv (Cornell University)|Oct 8, 2019

Sparse and Compressive Sensing Techniques27 references21 citations

TL;DR

This paper proposes a sparse scattering network that replaces complex learned layers with a single learned dictionary applied to scattering coefficients, achieving higher ImageNet top-1 accuracy (83.7%) than AlexNet by reducing intra-class variability through positive ℓ¹ sparse coding. The method uses homotopy-based iterative thresholding for fast, exponentially convergent sparse coding within a deep convolutional architecture.

ABSTRACT

We introduce a sparse scattering deep convolutional neural network, which provides a simple model to analyze properties of deep representation learning for classification. Learning a single dictionary matrix with a classifier yields a higher classification accuracy than AlexNet over the ImageNet 2012 dataset. The network first applies a scattering transform that linearizes variabilities due to geometric transformations such as translations and small deformations. A sparse $\\ell^1$ dictionary coding reduces intra-class variability while preserving class separation through projections over unions of linear spaces. It is implemented in a deep convolutional network with a homotopy algorithm having an exponential convergence. A convergence proof is given in a general framework that includes ALISTA. Classification results are analyzed on ImageNet.

Motivation & Objective

• To simplify deep representation learning by reducing complex networks to a single learned dictionary matrix.
• To analyze how dictionary learning improves scattering representations for image classification.
• To develop a mathematically tractable deep network architecture with provable convergence for sparse coding.
• To achieve high classification accuracy on ImageNet using only one learned dictionary, avoiding multi-layered learning blocks.

Proposed method

• The network applies a predefined scattering transform to linearize geometric variations such as translations and small deformations.
• Scattering coefficients are projected via a learned dictionary matrix D to compute a positive ℓ¹ sparse code α.
• A homotopy iterative soft-thresholding algorithm (ISTC) is used for sparse coding, with exponential convergence proven under general conditions.
• The dictionary D and classifier are jointly optimized via stochastic gradient descent to minimize classification loss.
• The method uses a simplified deep convolutional architecture where the dictionary is learned end-to-end with the classifier.
• The algorithm is implemented using ReLU activations and leverages the structure of ALISTA and homotopy continuation for fast convergence.

Experimental results

Research questions

• RQ1 Can a single learned dictionary matrix significantly improve the performance of a predefined scattering representation on ImageNet?
• RQ2 Does homotopy-based sparse coding with exponential convergence enable better optimization in deep networks compared to standard ISTA or FISTA?
• RQ3 How does sparse coding over scattering coefficients reduce intra-class variability while preserving inter-class separability?
• RQ4 To what extent can a deep network be reduced to a single dictionary learning problem without sacrificing accuracy?
• RQ5 What mathematical conditions ensure exponential convergence of the homotopy sparse coding algorithm in this framework?

Key findings

• The proposed sparse scattering network achieves a top-1 accuracy of 83.7% on ImageNet 2012, surpassing AlexNet’s performance.
• Using a single learned dictionary D with positive ℓ¹ sparse coding reduces the approximation error of the scattering representation by eliminating nearly half the energy, interpreted as non-informative clutter.
• The homotopy ISTC algorithm with W = D achieves exponential convergence in practice, even when theoretical incoherence conditions are not satisfied.
• After 12 iterations, ISTC reduces the relative mean square error to 0.02, outperforming FISTA (0.23) and ISTA (0.45) in sparse code approximation.
• Replacing the full scattering representation LSx with the sparse code Dα¹ improves classification accuracy by 14% compared to using α¹ directly.
• The learned dictionary vectors Dm act as discriminative features in the scattering coefficient space, optimized to preserve class-discriminative directions.