[Paper Review] Bagging in overparameterized learning: Risk characterization and risk monotonization

Pratik Patil, Jin‐Hong Du+1 more|arXiv (Cornell University)|Oct 20, 2022

Statistical Methods and Inference4 citations

TL;DR

This paper proposes a cross-validation framework to optimize bagged predictors in overparameterized regimes, characterizing the asymptotic prediction risk of subagging and splagging variants under proportional asymptotics. It proves that cross-validated bagging monotonizes the risk profile, eliminating double descent behavior while improving generalization beyond full-data training.

ABSTRACT

Bagging is a commonly used ensemble technique in statistics and machine learning to improve the performance of prediction procedures. In this paper, we study the prediction risk of variants of bagged predictors under the proportional asymptotics regime, in which the ratio of the number of features to the number of observations converges to a constant. Specifically, we propose a general strategy to analyze the prediction risk under squared error loss of bagged predictors using classical results on simple random sampling. Specializing the strategy, we derive the exact asymptotic risk of the bagged ridge and ridgeless predictors with an arbitrary number of bags under a well-specified linear model with arbitrary feature covariance matrices and signal vectors. Furthermore, we prescribe a generic cross-validation procedure to select the optimal subsample size for bagging and discuss its utility to eliminate the non-monotonic behavior of the limiting risk in the sample size (i.e., double or multiple descents). In demonstrating the proposed procedure for bagged ridge and ridgeless predictors, we thoroughly investigate the oracle properties of the optimal subsample size and provide an in-depth comparison between different bagging variants.

Motivation & Objective

• To characterize the asymptotic prediction risk of bagged ridge and ridgeless estimators under proportional asymptotics.
• To develop a generic cross-validation procedure that selects optimal subsample size to monotonize the limiting risk profile.
• To compare subagging (with replacement) and splagging (without replacement) in terms of risk performance and monotonicity.
• To establish theoretical guarantees for risk monotonization under the proposed framework.
• To demonstrate the oracle properties of the optimal subsample size across different bagging variants.

Proposed method

• Uses classical results from simple random sampling to derive exact asymptotic risk expressions for bagged predictors.
• Applies cross-validation to select the optimal subsample size by minimizing estimated prediction risk on held-out data.
• Analyzes both subagging (with replacement) and splagging (without replacement) as distinct bagging variants.
• Derives risk expressions as functions of the aspect ratio φ = p/n, feature covariance, and signal strength.
• Employs a proportional asymptotic regime where n, p → ∞ with p/n → φ.
• Validates theoretical findings via extensive simulations across multiple models (M-ISO-LI, M-AR1-LI) with varying SNR and feature structures.

Experimental results

Research questions

• RQ1 What is the exact asymptotic prediction risk of subagging and splagging ridge and ridgeless estimators in overparameterized linear models?
• RQ2 Can a cross-validation procedure be designed to provably monotonize the asymptotic risk as a function of the sample size or aspect ratio?
• RQ3 How do subagging and splagging compare in terms of risk performance and robustness to model mis-specification?
• RQ4 Does the optimal subsample size selected via cross-validation achieve oracle-like performance in finite samples?
• RQ5 Under what conditions does bagging eliminate the double descent phenomenon in generalization error?

Key findings

• The proposed cross-validation framework successfully monotonizes the asymptotic risk of bagged predictors, eliminating non-monotonic behavior such as double descent.
• Subagging with multiple bags consistently reduces prediction risk compared to unbagged predictors, especially in high-dimensional regimes.
• For ridgeless and ridge predictors, the optimal subsample size selected via cross-validation achieves risk performance close to the theoretical oracle.
• The asymptotic risk of splagging (without replacement) is generally lower than subagging (with replacement) under the same conditions, particularly when M is large.
• In simulations, the cross-validated bagged predictor outperforms the full-data estimator across all SNR levels and feature correlation structures.
• The risk curves for cross-validated bagged predictors are monotonic in the aspect ratio φ, confirming theoretical predictions.