[Paper Review] Optimal Rates for Learning with Nyström Stochastic Gradient Methods

Junhong Lin, Lorenzo Rosasco|arXiv (Cornell University)|Oct 21, 2017

Stochastic Gradient Optimization Techniques29 references3 citations

TL;DR

This paper proposes Nyström Stochastic Gradient Methods (NySGM), a novel algorithm combining stochastic gradient descent with Nyström subsampling for nonparametric regression. By leveraging mini-batches and adaptive step-sizes, NySGM achieves optimal learning rates with reduced computational cost—O(n^{1.5}) time and O(n^{1.5}) memory—compared to O(n^2) for Nyström KRR, while matching the statistical performance of state-of-the-art methods.

ABSTRACT

In the setting of nonparametric regression, we propose and study a combination of stochastic gradient methods with Nyström subsampling, allowing multiple passes over the data and mini-batches. Generalization error bounds for the studied algorithm are provided. Particularly, optimal learning rates are derived considering different possible choices of the step-size, the mini-batch size, the number of iterations/passes, and the subsampling level. In comparison with state-of-the-art algorithms such as the classic stochastic gradient methods and kernel ridge regression with Nyström, the studied algorithm has advantages on the computational complexity, while achieving the same optimal learning rates. Moreover, our results indicate that using mini-batches can reduce the total computational cost while achieving the same optimal statistical results.

Motivation & Objective

• To develop a scalable learning algorithm for nonparametric regression that maintains optimal statistical rates while reducing computational cost.
• To integrate Nyström subsampling with stochastic gradient methods to avoid full kernel matrix computation.
• To analyze the generalization error and derive optimal convergence rates under various parameter choices.
• To demonstrate that mini-batch usage can reduce total computational cost without sacrificing statistical performance.
• To provide theoretical guarantees using concentration inequalities, operator theory, and convex analysis.

Proposed method

• The algorithm uses stochastic gradient updates with mini-batches drawn uniformly at random from the training data.
• At each iteration, the solution is projected onto a subspace defined by a Nyström-approximated kernel matrix using a subset of landmark points.
• The method employs a time-varying step-size η_t = η_1 t^{-θ} with θ ∈ [0,1) to balance convergence and stability.
• The subsampling level, number of iterations, step-size, and mini-batch size are jointly optimized to achieve optimal learning rates.
• Theoretical analysis relies on concentration inequalities, operator norms, and properties of reproducing kernel Hilbert spaces (RKHS).
• The method is analyzed under both fixed and random design settings, with error bounds derived for generalization error.

Experimental results

Research questions

• RQ1 Can stochastic gradient methods with Nyström subsampling achieve optimal learning rates in nonparametric regression?
• RQ2 How do the choices of step-size, mini-batch size, number of iterations, and subsampling level affect the generalization error?
• RQ3 Does using mini-batches in Nyström-based SGD reduce total computational cost while preserving optimal statistical performance?
• RQ4 Can the proposed algorithm match the optimal error bounds of kernel ridge regression with significantly lower computational complexity?
• RQ5 What theoretical guarantees can be established for the generalization error using concentration and operator theory?

Key findings

• NySGM achieves optimal learning rates of O(n^{-0.5}) after one pass over the data, even without benign assumptions on the problem.
• The algorithm requires O(n^{1.5}) time and O(n^{1.5}) memory, significantly reducing the O(n^2) cost of standard Nyström KRR.
• Using mini-batches reduces total computational cost while maintaining the same optimal statistical results, a counterintuitive advantage over classic SGM.
• Theoretical error bounds are derived as O((ηt)^{-(2ζ+1)}) + O(n^{-(2ζ+1)/(2ζ+γ+1)}) for ζ ≤ 1/2, with high probability.
• With approximate leverage score sampling for landmark points, the method maintains optimal rates under mild conditions on the subsampling level m ≳ n^{γ/(2ζ+γ+1)} log n.
• Cross-validation tuning of the step-size achieves error bounds within a logarithmic factor of the optimal rate, with high probability.