[Paper Review] Rivalry of Two Families of Algorithms for Memory-Restricted Streaming PCA

Chunliang Li, Hsuan-Tien Lin+1 more|arXiv (Cornell University)|Jun 4, 2015

Stochastic Gradient Optimization Techniques12 references21 citations

TL;DR

This paper proposes DBPCA, a dynamic block power method for memory-restricted streaming PCA that automatically adjusts block sizes to accelerate convergence and improve stability. It provides the first convergence rate analysis for stochastic gradient descent (SPCA) in the general k>1 case without requiring a pre-specified error goal, and empirically shows DBPCA significantly outperforms both SPCA and BPCA in spectral error reduction with greater robustness to hyperparameter choices.

ABSTRACT

We study the problem of recovering the subspace spanned by the first $k$ principal components of $d$-dimensional data under the streaming setting, with a memory bound of $O(kd)$. Two families of algorithms are known for this problem. The first family is based on the framework of stochastic gradient descent. Nevertheless, the convergence rate of the family can be seriously affected by the learning rate of the descent steps and deserves more serious study. The second family is based on the power method over blocks of data, but setting the block size for its existing algorithms is not an easy task. In this paper, we analyze the convergence rate of a representative algorithm with decayed learning rate (Oja and Karhunen, 1985) in the first family for the general $k>1$ case. Moreover, we propose a novel algorithm for the second family that sets the block sizes automatically and dynamically with faster convergence rate. We then conduct empirical studies that fairly compare the two families on real-world data. The studies reveal the advantages and disadvantages of these two families.

Motivation & Objective

• To analyze the convergence rate of stochastic gradient descent (SPCA) for streaming PCA in the general k>1 case without pre-specifying an error goal.
• To address the challenge of manual block size selection in block power methods, which affects convergence and stability.
• To propose DBPCA, a novel dynamic block power method that automatically adjusts block sizes to improve convergence speed and stability.
• To fairly compare DBPCA with SPCA and BPCA on real-world data, focusing on spectral error reduction under O(kd) memory constraints.

Proposed method

• Proposes a convergence rate analysis for the classic SPCA algorithm (Oja and Karhunen, 1985) in the k>1 case, establishing theoretical guarantees without requiring a pre-specified error target.
• Introduces DBPCA, a dynamic block power method that increases block size over time to balance early improvement and long-term convergence.
• Employs a probabilistic update rule in DBPCA that ensures error reduction with high probability at each block, based on estimated eigenvalue gaps.
• Uses a block-based approximation of the empirical covariance matrix product to mimic the power method, enabling memory efficiency.
• Dynamically selects block sizes using a threshold based on the estimated spectral gap and error decay, avoiding manual tuning.
• Employs a spectral error metric as the primary evaluation criterion, which is more sensitive to subspace accuracy than reconstruction error.

Experimental results

Research questions

• RQ1 Can the convergence rate of SPCA be theoretically analyzed for k>1 without requiring a pre-specified error goal?
• RQ2 How does the performance of SPCA depend on the learning rate parameter c, and can it be made more robust to hyperparameter choice?
• RQ3 Can a block power method be designed to automatically adjust block sizes to improve convergence speed and stability?
• RQ4 How do DBPCA and SPCA compare empirically in terms of spectral error reduction across real-world datasets?
• RQ5 What is the impact of block size selection on the convergence behavior and stability of block power methods in streaming PCA?

Key findings

• DBPCA significantly outperforms BPCA in 6 out of 8 experimental cases by t-test at 95% confidence, with faster and more stable error reduction.
• DBPCA converges to lower spectral error than SPCA in 6 out of 8 cases, demonstrating superior convergence rate and robustness.
• SPCA is highly sensitive to the learning rate parameter c, with performance varying drastically across different values and k settings.
• BPCA's performance is highly dependent on block size L, with suboptimal choices leading to slow convergence or stagnation, and results are less stable over runs.
• DBPCA achieves stable error reduction after each update, consistent with its theoretical guarantee of error reduction with high probability.
• The empirical results suggest that the theoretical lower bound on γ² may be improvable, as larger values of γ² yield better performance in practice.