[Paper Review] Ridge Regression and Provable Deterministic Ridge Leverage Score Sampling

Shannon McCurdy|arXiv (Cornell University)|Mar 15, 2018

Sparse and Compressive Sensing Techniques8 references3 citations

TL;DR

This paper introduces a provably accurate deterministic algorithm for column sampling using ridge leverage scores, ensuring $(1+\epsilon)$-approximation guarantees for column subset selection, projection-cost preservation, and spectral bounds. It demonstrates that under power-law decay of ridge leverage scores, the method matches randomized algorithms in accuracy while offering interpretability through deterministic, repeatable results.

ABSTRACT

Ridge leverage scores provide a balance between low-rank approximation and regularization, and are ubiquitous in randomized linear algebra and machine learning. Deterministic algorithms are also of interest in the moderately big data regime, because deterministic algorithms provide interpretability to the practitioner by having no failure probability and always returning the same results. We provide provable guarantees for deterministic column sampling using ridge leverage scores. The matrix sketch returned by our algorithm is a column subset of the original matrix, yielding additional interpretability. Like the randomized counterparts, the deterministic algorithm provides (1 + ε) error column subset selection, (1 + ε) error projection-cost preservation, and an additive-multiplicative spectral bound. We also show that under the assumption of power-law decay of ridge leverage scores, this deterministic algorithm is provably as accurate as randomized algorithms. Lastly, ridge regression is frequently used to regularize ill-posed linear least-squares problems. While ridge regression provides shrinkage for the regression coefficients, many of the coefficients remain small but non-zero. Performing ridge regression with the matrix sketch returned by our algorithm and a particular regularization parameter forces coefficients to zero and has a provable (1 + ε) bound on the statistical risk. As such, it is an interesting alternative to elastic net regularization.

Motivation & Objective

• To develop a deterministic algorithm for column sampling that ensures high accuracy and interpretability in moderately large datasets.
• To provide theoretical guarantees for column subset selection, projection-cost preservation, and spectral approximation using ridge leverage scores.
• To show that deterministic ridge leverage score sampling performs as well as randomized methods under power-law decay of leverage scores.
• To demonstrate that ridge regression on the deterministic sketch yields a $(1+\epsilon)$-bound on statistical risk, offering a viable alternative to elastic net regularization.

Proposed method

• The algorithm selects columns based on deterministic ridge leverage scores, defined as $\bar{\tau}_i(\mathbf{A}) = \mathbf{a}_i^T(\mathbf{A}\mathbf{A}^T + \lambda_2\mathbf{I})^+\mathbf{a}_i$, with $\lambda_2 = \frac{1}{k}\|\mathbf{A} - \mathbf{A}_k\|_F^2$.
• It selects a column subset $\mathbf{C}$ such that the sum of selected ridge leverage scores exceeds $\bar{t} - \epsilon$, where $\bar{t}$ is the total trace of the ridge leverage score matrix.
• The method ensures $(1+\epsilon)$-approximation for column subset selection, projection-cost preservation, and spectral norm bounds via analytical derivation and matrix perturbation theory.
• It leverages power-law decay assumptions on ridge leverage scores to bound the number of columns required, showing equivalence to randomized sampling in accuracy.
• The algorithm uses a rank-$k$ SVD approximation $\mathbf{A}_k$ to define the regularization parameter $\lambda_2$, ensuring stability and theoretical guarantees.
• Theoretical bounds are derived using matrix concentration inequalities and spectral decomposition, with proofs adapted from Papailiopoulos et al. (2014) and Cohen et al. (2017).

Experimental results

Research questions

• RQ1 Can a deterministic algorithm for ridge leverage score sampling achieve $(1+\epsilon)$-approximation guarantees for column subset selection, projection-cost preservation, and spectral bounds?
• RQ2 Under what conditions does deterministic ridge leverage score sampling match the accuracy of randomized sampling?
• RQ3 How does ridge regression on the deterministic sketch compare to elastic net in terms of statistical risk and coefficient sparsity?
• RQ4 What is the theoretical bound on the number of columns required when ridge leverage scores follow a power-law decay?
• RQ5 Can the deterministic algorithm provide interpretability without sacrificing accuracy in moderately large datasets?

Key findings

• The deterministic algorithm achieves $(1+\epsilon)$-approximation for column subset selection, projection-cost preservation, and spectral norm bounds, matching the guarantees of randomized ridge leverage score sampling.
• Under power-law decay of ridge leverage scores, the deterministic algorithm requires only $O(k \log k / \epsilon^2)$ columns, matching the sample complexity of randomized methods.
• The algorithm ensures a $(1+\epsilon)$-bound on statistical risk when performing ridge regression on the sketch, with coefficients driven to zero due to regularization.
• The method outperforms classical leverage score sampling by avoiding reliance on small singular values, providing more stable and interpretable results.
• Empirical results on LGG tumor multi-omic data show that only a small number of columns (e.g., ~100) are needed to maintain low error, even with $k=3$.
• Histograms of prediction differences ($\hat{\mathbf{y}}_{\mathbf{A}} - \hat{\mathbf{y}}_{\mathbf{C}}$) show minimal deviation, confirming strong predictive fidelity of the sketch.