[Paper Review] Ridge Regression: Structure, Cross-Validation, and Sketching

Sifan Liu, Edgar Dobriban|arXiv (Cornell University)|Oct 6, 2019

Sparse and Compressive Sensing Techniques29 references17 citations

TL;DR

This paper provides a unified analysis of ridge regression in high-dimensional asymptotic regimes, deriving a precise estimator structure as a covariance-dependent linear combination of true parameters and noise. It proposes a bias-corrected cross-validation method for regularization parameter selection and demonstrates that primal and dual sketching methods offer significant computational speedups with minimal accuracy loss—sometimes reducing error by only 5% while halving computation time.

ABSTRACT

We study the following three fundamental problems about ridge regression: (1) what is the structure of the estimator? (2) how to correctly use cross-validation to choose the regularization parameter? and (3) how to accelerate computation without losing too much accuracy? We consider the three problems in a unified large-data linear model. We give a precise representation of ridge regression as a covariance matrix-dependent linear combination of the true parameter and the noise. We study the bias of $K$-fold cross-validation for choosing the regularization parameter, and propose a simple bias-correction. We analyze the accuracy of primal and dual sketching for ridge regression, showing they are surprisingly accurate. Our results are illustrated by simulations and by analyzing empirical data.

Motivation & Objective

• To understand the structural form of the ridge regression estimator in high-dimensional asymptotic models.
• To address the bias in K-fold cross-validation for regularization parameter selection and propose a correction.
• To analyze the accuracy and computational efficiency of primal and dual sketching methods in ridge regression.
• To establish theoretical guarantees for sketching accuracy under random matrix theory frameworks.
• To provide practical insights for choosing regularization parameters and accelerating computation without sacrificing predictive performance.

Proposed method

• Derives a precise representation of ridge regression as a linear combination of the true parameter and noise, with scaling matrices dependent on the population covariance structure.
• Proposes a downward scaling bias correction for K-fold cross-validation by analytically characterizing its bias in the asymptotic regime.
• Analyzes primal and dual sketching via random matrix theory, modeling the effect of sketching on the sample covariance matrix.
• Uses free probability and asymptotic random matrix theory to compute limits of trace expressions involving sample covariance matrices.
• Employs the Marchenko-Pastur law and free additive convolution to characterize the behavior of sketched estimators.
• Validates theoretical findings through simulations and empirical data analysis, comparing MSE, bias, and variance across methods.

Experimental results

Research questions

• RQ1 How is the ridge regression estimator structured in terms of the true parameter and noise, given the population covariance matrix?
• RQ2 What is the nature and magnitude of the bias in K-fold cross-validation when selecting the ridge regularization parameter?
• RQ3 How accurate are primal and dual sketching methods for ridge regression in high-dimensional settings?
• RQ4 Can sketching methods reduce computational cost while maintaining or even improving prediction accuracy?
• RQ5 What is the optimal trade-off between bias, variance, and computational efficiency in sketched ridge regression?

Key findings

• The ridge estimator is well-approximated by a linear transformation of the true parameter, with scaling determined by the population covariance matrix and regularization parameter.
• Cross-validation exhibits a systematic downward bias in estimating the optimal regularization parameter, which is corrected by a simple downward scaling of the CV-selected value.
• Primal and dual sketching methods achieve up to 50% reduction in computational cost with only 5% increase in mean squared error (MSE) in some settings.
• Surprisingly, sketching can improve MSE when the original regularization parameter is suboptimal, due to implicit bias correction.
• Dual sketching reduces variance more effectively than primal sketching, but at the cost of higher bias; the choice depends on the relative importance of bias vs. variance.
• The optimal regularization parameter for sketched ridge regression is generally lower than for standard ridge, and the performance gap between primal and dual sketching depends on the sketching dimension relative to sample size.