[Paper Review] On Fast Leverage Score Sampling and Optimal Learning

Alessandro Rudi, Daniele Calandriello+2 more|arXiv (Cornell University)|Oct 31, 2018

Stochastic Gradient Optimization Techniques28 references15 citations

TL;DR

This paper introduces BLESS, a novel algorithm for fast and accurate leverage score sampling in kernel methods, leveraging a coarse-to-fine strategy with adaptive regularization paths. It achieves state-of-the-art computational efficiency and accuracy for kernel ridge regression, reducing time and space complexity to $ tilde{ cal{O}}(n d_{ ext{eff}})$ and $ tilde{ cal{O}}(d_{ ext{eff}}^2)$, respectively, where $d_{ ext{eff}}$ is the effective dimension, significantly improving upon uniform sampling methods.

ABSTRACT

Leverage score sampling provides an appealing way to perform approximate computations for large matrices. Indeed, it allows to derive faithful approximations with a complexity adapted to the problem at hand. Yet, performing leverage scores sampling is a challenge in its own right requiring further approximations. In this paper, we study the problem of leverage score sampling for positive definite matrices defined by a kernel. Our contribution is twofold. First we provide a novel algorithm for leverage score sampling and second, we exploit the proposed method in statistical learning by deriving a novel solver for kernel ridge regression. Our main technical contribution is showing that the proposed algorithms are currently the most efficient and accurate for these problems.

Motivation & Objective

• To address the computational bottleneck of exact leverage score computation in large-scale kernel methods.
• To develop a fast, accurate, and non-distributed algorithm for approximate leverage score sampling in positive definite matrices.
• To improve the computational efficiency of kernel ridge regression by replacing uniform sampling with leverage score sampling.
• To achieve optimal learning bounds with reduced time and memory complexity proportional to the effective dimension $d_{ ext{eff}}$.
• To provide theoretical guarantees on approximation accuracy and computational complexity for the proposed method.

Proposed method

• Proposes BLESS, a coarse-to-fine algorithm that alternates between uniform and leverage score sampling on progressively larger sets.
• Uses a sequence of decreasing regularization parameters $\lambda_h$ to gradually refine leverage score estimates, ensuring multiplicative accuracy bounds.
• Employs intermediate uniform sampling sets $U_h$ of size proportional to $1/\lambda_h$ to compute approximate leverage scores efficiently.
• Introduces BLESS-R, a rejection sampling variant that avoids explicit intermediate sets and maintains the same complexity bounds.
• Leverages the fact that $M_h \propto d_{\text{eff}}(\lambda_h)$, ensuring the number of samples scales with the effective dimension.
• Uses preconditioned conjugate gradient methods in the learning phase to achieve optimal convergence rates.

Experimental results

Research questions

• RQ1 Can we achieve state-of-the-art accuracy in leverage score sampling without relying on distributed computing?
• RQ2 Can leverage score sampling reduce the time and space complexity of kernel ridge regression below that of uniform sampling?
• RQ3 What is the optimal trade-off between approximation accuracy and computational cost in leverage score sampling for kernel matrices?
• RQ4 How can we efficiently compute leverage scores in a sequential, adaptive manner without full eigendecomposition?
• RQ5 Can the effective dimension $d_{\text{eff}}$ serve as a natural complexity parameter for kernel methods, replacing $\sqrt{n}$?

Key findings

• The proposed BLESS algorithm achieves state-of-the-art accuracy and computational complexity for leverage score sampling without requiring distributed resources.
• The time complexity of the proposed kernel ridge regression solver is $\ttilde{\mathcal{O}}(n d_{\text{eff}})$, and space complexity is $\ttilde{\mathcal{O}}(d_{\text{eff}}^2)$, where $d_{\text{eff}} \leq \sqrt{n}$.
• The algorithm maintains multiplicative accuracy bounds of the form $\frac{1}{1+t}\ell(i,\lambda) \leq \widetilde{\ell}(i,\lambda) \leq (1+t)\ell(i,\lambda)$ with $t$ controlled by the regularization path.
• The use of intermediate uniform sampling sets $U_h$ of size $\propto 1/\lambda_h$ ensures sufficient accuracy while keeping computational cost low.
• BLESS-R achieves the same complexity bounds as BLESS but with simpler implementation via rejection sampling, avoiding explicit intermediate sets.
• Theoretical analysis confirms that the method achieves optimal learning bounds for kernel ridge regression, matching the best-known guarantees in the literature.