[Paper Review] Data-Driven Sample Average Approximation with Covariate Information

Rohit Kannan, Güzi̇n Bayraksan+1 more|arXiv (Cornell University)|Jul 27, 2022

Risk and Portfolio Optimization22 citations

TL;DR

The paper embeds prediction models using covariates into data-driven SAA for conditional stochastic programs, introducing ER-SAA and two leave-one-out variants with convergence guarantees and empirical validation.

ABSTRACT

We study optimization for data-driven decision-making when we have observations of the uncertain parameters within the optimization model together with concurrent observations of covariates. Given a new covariate observation, the goal is to choose a decision that minimizes the expected cost conditioned on this observation. We investigate three data-driven frameworks that integrate a machine learning prediction model within a stochastic programming sample average approximation (SAA) for approximating the solution to this problem. Two of the SAA frameworks are new and use out-of-sample residuals of leave-one-out prediction models for scenario generation. The frameworks we investigate are flexible and accommodate parametric, nonparametric, and semiparametric regression techniques. We derive conditions on the data generation process, the prediction model, and the stochastic program under which solutions of these data-driven SAAs are consistent and asymptotically optimal, and also derive convergence rates and finite sample guarantees. Computational experiments validate our theoretical results, demonstrate the potential advantages of our data-driven formulations over existing approaches (even when the prediction model is misspecified), and illustrate the benefits of our new data-driven formulations in the limited data regime.

Motivation & Objective

• Motivate data-driven decision-making when covariates inform the distribution of uncertain parameters.
• Develop data-driven SAA frameworks that leverage regression predictions and residuals.
• Establish theoretical guarantees (consistency, asymptotic optimality, rates) for the proposed methods.
• Introduce and analyze jackknife-based variants to improve performance in limited data.
• Demonstrate applicability across parametric, nonparametric, and semiparametric regression settings.

Proposed method

• Model Y as Y = f*(X) + Q*(X)ε with covariates X and random error ε.
• Define and compare FI-SAA, ER-SAA, and two jackknife-based SAA variants for scenario generation.
• Use regression to estimate f* and Q*, and construct residual-based scenarios as c(z, projection of f-hat(X) + Q-hat(X)ε-hat).
• Incorporate projection onto the Y-support to ensure feasible scenarios and discuss optional denoising via jackknife corrections.
• Provide analysis showing convergence, rates, and finite-sample guarantees under mild assumptions; cover two-stage LP as running example.
• Demonstrate flexibility to use parametric, nonparametric, and semiparametric regression techniques (e.g., OLS, Lasso, kNN, RF).

Experimental results

Research questions

• RQ1 How can covariate information be used to approximate the conditional stochastic program solution?
• RQ2 When do ER-SAA and its jackknife variants yield asymptotically optimal and consistent solutions?
• RQ3 What are the convergence rates and finite-sample guarantees for data-driven SAA with covariates?
• RQ4 How do parametric versus nonparametric regression choices affect theoretical guarantees and practical performance?
• RQ5 Can the framework accommodate heteroscedastic error structures and still deliver tractable solutions?

Key findings

• ER-SAA achieves asymptotic optimality and convergence guarantees under mild assumptions.
• Leave-one-out residual variants (J-SAA and J+-SAA) offer potential improvements in small-sample regimes.
• The framework supports a broad class of prediction models, including OLS, Lasso, kNN, and RF, with heteroscedastic considerations.
• Projection onto the Y-support helps keep scenarios feasible without sacrificing theoretical guarantees.
• Empirical experiments validate theoretical results and show gains over existing approaches, even under model misspecification.