[Paper Review] Learning the Hierarchical Parts of Objects by Deep Non-Smooth Nonnegative Matrix Factorization

Jinshi Yu, Guoxu Zhou+2 more|arXiv (Cornell University)|Mar 20, 2018

Advanced Image and Video Retrieval Techniques37 references3 citations

TL;DR

This paper proposes Deep Non-Smooth Nonnegative Matrix Factorization (dnsNMF), a hierarchical deep learning framework that extends non-smooth NMF (nsNMF) by stacking multiple layers to learn both parts-based and increasingly abstract features. By integrating the nonnegativity and sparsity control of nsNMF with a deep architecture, dnsNMF achieves superior clustering performance on face datasets while maintaining data fidelity.

ABSTRACT

Nonsmooth Nonnegative Matrix Factorization (nsNMF) is capable of producing more localized, less overlapped feature representations than other variants of NMF while keeping satisfactory fit to data. However, nsNMF as well as other existing NMF methods is incompetent to learn hierarchical features of complex data due to its shallow structure. To fill this gap, we propose a deep nsNMF method coined by the fact that it possesses a deeper architecture compared with standard nsNMF. The deep nsNMF not only gives parts-based features due to the nonnegativity constraints, but also creates higher-level, more abstract features by combing lower-level ones. The in-depth description of how deep architecture can help to efficiently discover abstract features in dnsNMF is presented. And we also show that the deep nsNMF has close relationship with the deep autoencoder, suggesting that the proposed model inherits the major advantages from both deep learning and NMF. Extensive experiments demonstrate the standout performance of the proposed method in clustering analysis.

Motivation & Objective

• To address the limitation of shallow nsNMF in learning hierarchical, abstract features of complex data.
• To overcome the trade-off between sparsity and data fitting inherent in traditional NMF variants.
• To develop a deep architecture that preserves the nonnegativity and sparsity advantages of nsNMF while enabling multi-level abstraction.
• To establish a theoretical and empirical link between dnsNMF and deep autoencoders.
• To demonstrate the effectiveness of deepening the nsNMF structure for unsupervised representation learning.

Proposed method

• Proposes a deep architecture composed of multiple stacked layers of non-smooth NMF (nsNMF), where each layer applies the nsNMF model with a smoothing matrix S.
• Uses a hierarchical decomposition: X ≈ Z₁(S₁Z₂(S₂…ZₖHₖ)…), enabling multi-level feature abstraction.
• Employs a multiplicative update rule for all factor matrices (Zᵢ, Hᵢ, Sᵢ) to ensure nonnegativity and convergence.
• Introduces a Lipschitz continuity proof for the gradient of the objective function, supporting stable optimization.
• Establishes a theoretical equivalence between dnsNMF and a restricted deep autoencoder, linking it to deep learning principles.
• Employs a greedy layer-wise pre-training strategy to initialize parameters and improve optimization.

Experimental results

Research questions

• RQ1 Can a deep architecture enhance nsNMF’s ability to learn hierarchical, abstract features beyond low-level parts?
• RQ2 How does the depth of the network affect the clustering performance and feature abstraction in dnsNMF?
• RQ3 What is the relationship between dnsNMF and deep autoencoders, and does this connection explain its representational power?
• RQ4 Can dnsNMF maintain high data fitting accuracy while achieving high sparsity in both feature and representation matrices?
• RQ5 What is the optimal depth for balancing clustering accuracy and computational efficiency in real-world applications?

Key findings

• dnsNMF significantly improves clustering accuracy on AT&T ORL, JAFFE, and Yale face datasets compared to standard and shallow nsNMF.
• Clustering accuracy increases with depth up to a point (e.g., depth 3–5), after which performance plateaus, indicating diminishing returns beyond moderate depth.
• The model achieves better sparsity and less overlapping features than standard NMF and other variants, due to the non-smooth regularization via the smoothing matrix S.
• Theoretical analysis confirms that the gradient of the objective function in dnsNMF is Lipschitz continuous, supporting stable optimization.
• dnsNMF is mathematically equivalent to a restricted deep autoencoder, suggesting it inherits the benefits of both deep learning and NMF.
• Experiments show that dnsNMF outperforms existing deep NMF methods such as N-NMF, deep semi-NMF, and SDNMF in clustering tasks.