[Paper Review] Stochastic Chebyshev Gradient Descent for Spectral Optimization

In‐Su Han, Haim Avron+1 more|arXiv (Cornell University)|Feb 18, 2018

Stochastic Gradient Optimization Techniques7 citations

TL;DR

This paper proposes Stochastic Chebyshev Gradient Descent (SCGD), an unbiased stochastic gradient method for spectral optimization problems involving spectral-sum functions—such as log-determinant and nuclear norm—by combining randomized trace estimation with variance-optimal stochastic truncation of Chebyshev expansions. The key contribution is a provably optimal degree distribution that minimizes estimator variance, enabling faster and more stable convergence than prior biased or suboptimal stochastic methods, with empirical validation showing up to sixfold speedups in Gaussian process learning.

ABSTRACT

A large class of machine learning techniques requires the solution of optimization problems involving spectral functions of parametric matrices, e.g. log-determinant and nuclear norm. Unfortunately, computing the gradient of a spectral function is generally of cubic complexity, as such gradient descent methods are rather expensive for optimizing objectives involving the spectral function. Thus, one naturally turns to stochastic gradient methods in hope that they will provide a way to reduce or altogether avoid the computation of full gradients. However, here a new challenge appears: there is no straightforward way to compute unbiased stochastic gradients for spectral functions. In this paper, we develop unbiased stochastic gradients for spectral-sums, an important subclass of spectral functions. Our unbiased stochastic gradients are based on combining randomized trace estimators with stochastic truncation of the Chebyshev expansions. A careful design of the truncation distribution allows us to offer distributions that are variance-optimal, which is crucial for fast and stable convergence of stochastic gradient methods. We further leverage our proposed stochastic gradients to devise stochastic methods for objective functions involving spectral-sums, and rigorously analyze their convergence rate. The utility of our methods is demonstrated in numerical experiments.

Motivation & Objective

• To address the high computational cost of exact gradients in spectral optimization, particularly for spectral-sum functions like log-determinant and nuclear norm.
• To develop unbiased stochastic gradients for spectral-sums, overcoming the limitation of existing biased estimators in stochastic optimization.
• To design a variance-optimal truncation distribution for Chebyshev expansions that minimizes gradient estimator variance, crucial for stable and fast convergence.
• To enable efficient large-scale spectral optimization by reducing full gradient computation while maintaining convergence guarantees.
• To empirically validate the method on real-world tasks such as matrix completion and Gaussian process regression.

Proposed method

• Proposes an unbiased stochastic gradient estimator for spectral-sums by randomly sampling the truncation degree in Chebyshev polynomial expansions.
• Combines randomized trace estimation with stochastic truncation to avoid full eigenvalue decompositions, reducing computational cost.
• Derives a variance-optimal degree distribution for the truncation that minimizes the variance of the gradient estimator, ensuring faster convergence.
• Leverages the optimal degree distribution to design two stochastic optimization frameworks: standard SGD and SVRG, both with rigorous convergence analysis.
• Uses Chebyshev polynomial approximation to express spectral functions as traces of matrix polynomials, enabling efficient stochastic trace estimation.
• Employs matrix perturbation bounds (e.g., via Chebyshev polynomial derivative inequalities) to control error propagation in the stochastic approximation.

Experimental results

Research questions

• RQ1 Can unbiased stochastic gradients be constructed for spectral-sum functions without relying on full eigenvalue decompositions?
• RQ2 What truncation distribution for Chebyshev expansions minimizes the variance of the stochastic gradient estimator?
• RQ3 How does the proposed variance-optimal distribution compare to standard choices (e.g., uniform or geometric) in terms of convergence speed and stability?
• RQ4 Can the proposed method achieve faster convergence than existing stochastic or deterministic methods in large-scale spectral optimization tasks?
• RQ5 What is the theoretical convergence rate of the proposed SGD and SVRG variants under the optimal degree distribution?

Key findings

• The proposed variance-optimal degree distribution reduces gradient estimator variance by an order of magnitude compared to uniform or geometric distributions, as shown in Figure 1.
• The SCGD method achieves sublinear convergence rate for SGD and linear convergence rate for SVRG, both critically dependent on the optimal degree distribution.
• In matrix completion and Gaussian process regression, the method runs up to six times faster than the state-of-the-art specialized method [8] on the Szeged humid dataset.
• The method maintains stable convergence due to the low variance of the stochastic gradients, which is essential for practical optimization.
• Theoretical analysis confirms that the convergence rates are tight and depend on the variance-minimizing truncation distribution, which is derived via optimization over the degree distribution.
• Empirical results demonstrate that the method scales efficiently to large-scale real-world problems, outperforming both general-purpose and specialized baselines.