[Paper Review] Cross-validation: what does it estimate and how well does it do it?

Stephen Bates, Trevor Hastie+1 more|arXiv (Cornell University)|Apr 1, 2021

Statistical Methods and Inference51 references45 citations

TL;DR

The paper shows that cross-validation estimates the average prediction error across training sets (Err) rather than the error of the specific final model (ErrXY) and introduces nested cross-validation to improve confidence interval coverage for prediction error.

ABSTRACT

Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. Ideally, one would like to think that cross-validation estimates the prediction error for the model at hand, fit to the training data. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population. We further show that this phenomenon occurs for most popular estimates of prediction error, including data splitting, bootstrapping, and Mallow's Cp. Next, the standard confidence intervals for prediction error derived from cross-validation may have coverage far below the desired level. Because each data point is used for both training and testing, there are correlations among the measured accuracies for each fold, and so the usual estimate of variance is too small. We introduce a nested cross-validation scheme to estimate this variance more accurately, and we show empirically that this modification leads to intervals with approximately correct coverage in many examples where traditional cross-validation intervals fail.

Motivation & Objective

• Clarify the estimand of cross-validation in predictive modeling, especially for linear models.
• Show that CV estimates Err (average error over training sets) rather than ErrXY (error of the final model).
• Demonstrate the limitations of naïve CV-based confidence intervals due to fold correlations.
• Propose nested cross-validation (NCV) to obtain more accurate prediction-error confidence intervals.
• Compare CV with related prediction-error estimators such as data splitting, Mallows Cp, and bootstrap.

Proposed method

• Define ErrXY as the out-of-sample error of the model fitted on the full training set and Err as the expectation over unseen training sets.
• Prove that linearly invariant estimators (including CV with OLS and squared error) are conditionally independent of ErrXY given X (Theorem 1).
• Establish that CV targets Err rather than ErrXY, and that ErrX (the conditional mean given X) is a closer population analogue to Err than ErrXY (Theorem 2 and Corollaries).
• Analyze the bias of CV due to using smaller training subsets in CV versus full-sample training (discuss bias in different regimes).
• Introduce nested cross-validation (NCV) as a modification to CV to yield unbiased standard-error estimates and better coverage for prediction-error intervals (Theorem 3).
• Relate CV and Cp-type estimators as linearly invariant estimators with similar target behavior (Section 3.5).

Figure 1 : A plot of the true error of a model versus the CV estimates for 1000 replicates of the model from Section 1.1 . The blue curve shows the average midpoint of the naïve CV confidence intervals. The green bands show the average 90% confidence interval for prediction error given by naïve CV.

Experimental results

Research questions

• RQ1 What precisely does cross-validation estimate in predictive modeling, particularly under linear models?
• RQ2 How do CV-based intervals perform in terms of coverage for predicting error, and can they be improved?
• RQ3 How do data-splitting, Mallows Cp, and bootstrap compare as estimators of prediction error?
• RQ4 Can a modified CV scheme (nested CV) provide more accurate confidence intervals for prediction error?
• RQ5 What is the relationship between Err, ErrX, ErrXY and the estimator produced by CV in high-dimensional/linear settings?

Key findings

• CV estimates Err (average prediction error across training sets) rather than ErrXY (error of the model fitted on the observed training set).
• Linearly invariant estimators like CV using OLS have lower MSE when estimating Err or ErrX than when estimating ErrXY (Theorem 1 and Corollary 1).
• ErrX and Err are asymptotically closer to Err than ErrXY in the proportional asymptotic regime, with Var(ErrX) = Θ(1/n^2) and Var(ErrXY|X) = Θ(1/n) (Theorem 2).
• Naïve CV intervals often undercover due to ignoring correlations among fold-errors; NCV provides intervals with approximately correct coverage in many cases (Figure 1 discussion and Theorem 3).
• Mallow’s Cp and Cp-based methods are linearly invariant and exhibit similar targeting of Err/ErrX rather than ErrXY (Section 3.5).

Figure 2 : Possible targets of inference for cross-validation. Here, $(X,Y)$ is the training data and $\mbox{{Err}}_{XY}$ is the average error of the model fit on $(X,Y)$ on a test data set of infinite size. From left to right, the random variables above are a constant, a function of $X$ only, and a