[Paper Review] Online Low-Rank Subspace Clustering by Basis Dictionary Pursuit

Jie Shen, Ping Li+1 more|arXiv (Cornell University)|Mar 28, 2015

Sparse and Compressive Sensing Techniques29 references44 citations

TL;DR

This paper proposes an online low-rank subspace clustering method via Basis Dictionary Pursuit that reduces memory cost from O(n²) to O(pd), where p is ambient dimension and d is estimated rank. The method achieves asymptotic convergence to a stationary point of the expected loss and demonstrates fast, robust, and memory-efficient performance on synthetic and real datasets.

ABSTRACT

Low-Rank Representation (LRR) has been a significant method for segmenting data that are generated from a union of subspaces. It is also known that solving LRR is challenging in terms of time complexity and memory footprint, in that the size of the nuclear norm regularized matrix is n-by-n (where n is the number of samples). In this paper, we thereby develop a novel online implementation of LRR that reduces the memory cost from O(n2) to O(pd), with p being the ambient dimension and d being some estimated rank (d ≤ p ≪ n). We also establish the theoretical guarantee that the sequence of solutions produced by our algorithm converges to a stationary point of the expected loss function asymptotically. Extensive experiments on synthetic and realistic datasets further substantiate that our algorithm is fast, robust and memory efficient.

Motivation & Objective

• Address the high time complexity and memory footprint of traditional Low-Rank Representation (LRR), which scales as O(n²) due to n×n matrix operations.
• Develop an online variant of LRR that dynamically processes data streams without storing the full data matrix.
• Reduce memory usage from O(n²) to O(pd), where d ≪ n is an estimated low rank and p is the ambient dimension.
• Establish theoretical convergence of the algorithm’s solution sequence to a stationary point of the expected loss function.
• Ensure scalability and robustness for large-scale and streaming data applications in subspace clustering.

Proposed method

• Formulate online LRR using a basis dictionary pursuit framework to represent data in a low-dimensional subspace.
• Maintain a compact dictionary of size O(pd) instead of storing the full n×n representation matrix.
• Update the dictionary incrementally as new data samples arrive, enabling online processing.
• Use nuclear norm regularization to promote low-rank structure in the representation matrix.
• Apply a greedy pursuit strategy to select basis vectors that best represent the current data subspace.
• Ensure theoretical convergence by showing that the sequence of solutions approaches a stationary point of the expected loss function.

Experimental results

Research questions

• RQ1 Can online LRR be designed to significantly reduce memory usage while preserving clustering accuracy?
• RQ2 Does the proposed online algorithm converge to a stationary point of the expected loss function?
• RQ3 How does the memory efficiency of the proposed method compare to batch LRR in large-scale settings?
• RQ4 Can the algorithm maintain robustness and speed when processing streaming or large-scale data?
• RQ5 What is the impact of the estimated rank d and ambient dimension p on the algorithm’s performance and memory footprint?

Key findings

• The proposed method reduces memory cost from O(n²) to O(pd), enabling scalability to large datasets where n is large.
• The algorithm achieves asymptotic convergence to a stationary point of the expected loss function.
• Extensive experiments show the method is significantly faster and more memory-efficient than batch LRR.
• The method maintains robust performance on both synthetic and real-world datasets under varying noise and subspace conditions.
• The online nature of the algorithm allows for incremental processing without requiring full data storage.