[Paper Review] Bounding Optimality Gap in Stochastic Optimization via Bagging: Statistical Efficiency and Stability

Henry Lam, Huajie Qian|arXiv (Cornell University)|Oct 5, 2018

Risk and Portfolio OptimizationDecision Sciences33 references3 citations

TL;DR

This paper proposes a bagging-based method to estimate lower confidence bounds for the optimal value in stochastic optimization, leveraging resampled sample average approximations (SAA) to bound the optimality gap of a given solution. It establishes statistical efficiency and stability under limited data, with theoretical justification via symmetric statistics and kernel approximation, outperforming batching and existing methods in tightness and accuracy under non-smooth and high-dimensional settings.

ABSTRACT

We study a statistical method to estimate the optimal value, and the optimality gap of a given solution for stochastic optimization as an assessment of the solution quality. Our approach is based on bootstrap aggregating, or bagging, resampled sample average approximation (SAA). We show how this approach leads to valid statistical confidence bounds for non-smooth optimization. We also demonstrate its statistical efficiency and stability that are especially desirable in limited-data situations, and compare these properties with some existing methods. We present our theory that views SAA as a kernel in an infinite-order symmetric statistic, which can be approximated via bagging. We substantiate our theoretical findings with numerical results.

Motivation & Objective

• To develop a statistically efficient and stable method for estimating the optimal value and optimality gap in stochastic optimization when data are limited.
• To address the limitations of worst-case bounds from traditional SAA and stochastic approximation, which can be overly conservative for solution quality assessment.
• To improve upon existing statistical validation methods by leveraging data recycling through bagging to tighten confidence bounds.
• To provide theoretical justification for the method using symmetric statistics and kernel approximation, showing robustness and convergence properties.
• To empirically validate the method's performance across diverse problems, including non-smooth and high-dimensional cases.

Proposed method

• The method uses bootstrap aggregating (bagging) on resampled datasets to compute multiple SAA solutions, then averages their optimal values to form a lower confidence bound for the true optimal value.
• It treats the SAA as a kernel in an infinite-order symmetric statistic, which is approximated via bagging to ensure stability and reduce variance.
• The approach is grounded in duality theory and the rearrangement inequality, showing that the bagged estimator is downward biased but converges to the true optimal value with controlled error.
• Theoretical analysis establishes bias decay rates of order $ O((k^2/n)^{l+1} + k/n) $ for any fixed $ l \geq 0 $, indicating strong convergence under $ k = o(\sqrt{n}) $.
• The method is applied to non-smooth problems (e.g., CVaR, integer programs) and compared with batching, CRN, and other gap estimation techniques.
• Numerical experiments use independent samples for $ Z(\hat{x}) $ and resampled SAA for $ Z^* $, with confidence bounds derived via empirical quantiles or asymptotic approximations.

Experimental results

Research questions

• RQ1 Can bagging of resampled SAA solutions provide tighter and more stable confidence bounds for the optimal value in stochastic optimization than traditional batching?
• RQ2 How does the proposed bagging-based estimator compare in statistical efficiency and bias control to existing methods like SAA, CRN, or batch-based approaches?
• RQ3 What is the theoretical justification for the statistical properties (bias, variance, convergence) of the bagged SAA estimator under general non-smooth and high-dimensional settings?
• RQ4 To what extent does data recycling via bagging improve the trade-off between bound tightness and statistical accuracy in limited-data regimes?
• RQ5 Can the method be applied effectively to non-smooth problems such as CVaR or integer programs, where standard CLT-based methods fail?

Key findings

• The bagged SAA estimator achieves a bias decay rate of $ O((k^2/n)^{l+1} + k/n) $ for any fixed $ l \geq 0 $, indicating strong convergence under $ k = o(\sqrt{n}) $, which improves over batching.
• The method provides valid lower confidence bounds for $ Z^* $ and upper bounds for the optimality gap $ \mathcal{G}(\hat{x}) $, even in non-smooth and high-dimensional settings.
• Empirical results show that bagging consistently produces tighter and more stable confidence bounds than batching and other baseline methods across multiple problem types, including CVaR, integer programs, and portfolio optimization.
• The approach outperforms CRN and batch-based methods in terms of bound tightness and variance, especially when data are scarce.
• Theoretical analysis confirms that the bagged estimator is downward biased but converges to the true optimal value with controlled error, ensuring validity of confidence intervals.
• Numerical experiments demonstrate that the method maintains high stability and efficiency across varying data sizes and bootstrap iterations, with minimal sensitivity to model structure or smoothness.