[Paper Review] Large-Scale Methods for Distributionally Robust Optimization

Daniel Lévy, Yair Carmon+2 more|arXiv (Cornell University)|Oct 12, 2020

Risk and Portfolio OptimizationDecision Sciences60 references37 citations

TL;DR

The paper develops scalable stochastic gradient methods for distributionally robust optimization with CVaR and chi-squared uncertainty sets, achieving gradient-evaluation complexity independent of training size and parameter count, and introduces multi-level Monte Carlo gradient estimators for improved efficiency.

ABSTRACT

We propose and analyze algorithms for distributionally robust optimization of convex losses with conditional value at risk (CVaR) and $χ^2$ divergence uncertainty sets. We prove that our algorithms require a number of gradient evaluations independent of training set size and number of parameters, making them suitable for large-scale applications. For $χ^2$ uncertainty sets these are the first such guarantees in the literature, and for CVaR our guarantees scale linearly in the uncertainty level rather than quadratically as in previous work. We also provide lower bounds proving the worst-case optimality of our algorithms for CVaR and a penalized version of the $χ^2$ problem. Our primary technical contributions are novel bounds on the bias of batch robust risk estimation and the variance of a multilevel Monte Carlo gradient estimator due to [Blanchet & Glynn, 2015]. Experiments on MNIST and ImageNet confirm the theoretical scaling of our algorithms, which are 9--36 times more efficient than full-batch methods.

Motivation & Objective

• Address the need for scalable distributionally robust optimization (DRO) in machine learning and high-stakes decision settings.
• Develop gradient-based algorithms for CVaR and chi-squared (including penalized) uncertainty sets that do not scale with N or d.
• Provide theoretical guarantees on bias and variance of gradient estimators and establish optimal complexity bounds.
• Demonstrate practical improvements over full-batch methods on large-scale datasets like MNIST and ImageNet.

Proposed method

• Formulate DRO objectives for CVaR and chi-squared divergence (constrained and penalized variants).
• Use mini-batch based gradient estimators with bias bounds for the surrogate objective and variance bounds for the gradient.
• Apply Nesterov acceleration to exploit variance bounds and improve convergence.
• Introduce multi-level Monte Carlo (MLMC) gradient estimators to obtain unbiased gradient estimates with logarithmic sample complexity in batch size.
• Provide dual and primal-dual formulations to enable tractable optimization and complexity analyses.

Experimental results

Research questions

• RQ1 Can DRO objectives with CVaR and chi-squared uncertainty sets be optimized with gradient evaluations that do not depend on the training set size N or parameter count d?
• RQ2 What are the bias and variance properties of mini-batch and MLMC gradient estimators for these DRO objectives?
• RQ3 What are the resulting oracle complexity rates for achieving an epsilon-optimal solution under CVaR and chi-squared objectives?
• RQ4 Do the proposed methods scale in practice to large datasets (e.g., MNIST, ImageNet) and outperform full-batch approaches?

Key findings

• Proposed gradient methods achieve gradient-evaluation complexity independent of N and d for CVaR and chi-squared DRO objectives.
• Established bias bounds for mini-batch estimators: O(1/√n) for CVaR, O(1/n) for chi-squared penalized objectives, and O(1/n) under a Lipschitz inverse-CDF assumption.
• Showed variance of mini-batch gradient estimators decreases as O(1/n) for chi-squared bounded objectives, enabling efficient optimization.
• MLMC gradient estimators provide unbiased gradient estimates with logarithmic sample requirements, improving efficiency over naive mini-batch estimators.
• Empirical results on MNIST/ImageNet demonstrate 9–36x fewer gradient evaluations than full-batch methods while maintaining or improving robustness.