[Paper Review] Regularization via Mass Transportation

Soroosh Shafieezadeh-Abadeh, Daniel Kühn+1 more|arXiv (Cornell University)|Oct 27, 2017

Machine Learning and Algorithms75 references54 citations

TL;DR

The paper introduces distributionally robust learning with Wasserstein ambiguity sets around the empirical distribution, deriving tractable convex reformulations that yield new regularization effects and generalization guarantees, extendable to kernel methods and neural networks.

ABSTRACT

The goal of regression and classification methods in supervised learning is to minimize the empirical risk, that is, the expectation of some loss function quantifying the prediction error under the empirical distribution. When facing scarce training data, overfitting is typically mitigated by adding regularization terms to the objective that penalize hypothesis complexity. In this paper we introduce new regularization techniques using ideas from distributionally robust optimization, and we give new probabilistic interpretations to existing techniques. Specifically, we propose to minimize the worst-case expected loss, where the worst case is taken over the ball of all (continuous or discrete) distributions that have a bounded transportation distance from the (discrete) empirical distribution. By choosing the radius of this ball judiciously, we can guarantee that the worst-case expected loss provides an upper confidence bound on the loss on test data, thus offering new generalization bounds. We prove that the resulting regularized learning problems are tractable and can be tractably kernelized for many popular loss functions. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Motivation & Objective

• Motivate regularization as a worst-case risk under distributional ambiguity.
• Introduce Wasserstein-ball uncertainty sets around the empirical distribution to hedge against data perturbations.
• Develop tractable convex reformulations for linear, kernelized, and neural network models under this framework.
• Provide probabilistic interpretations of classical regularization via mass transportation and establish generalization guarantees.

Proposed method

• Define the Wasserstein ball around the empirical distribution to capture distributional uncertainty.
• Formulate a distributionally robust learning problem as a minimax objective over regression/classification losses.
• Prove tractable finite convex reformulations for linear hypotheses with piecewise affine losses or Lipschitz losses.
• Show kernelizable lifted formulations enabling nonlinear hypothesis spaces.
• Extend the approach to neural networks with convex regularization surrogates and stochastic proximal optimization.

Experimental results

Research questions

• RQ1 Can worst-case expected loss over a Wasserstein ball provide valid out-of-sample performance guarantees?
• RQ2 When does the distributionally robust objective admit tractable convex reformulations for common loss functions?
• RQ3 How can the framework be extended to nonlinear models, such as kernel methods and neural networks?
• RQ4 What probabilistic interpretations of classical regularization emerge from Wasserstein-based robustness?
• RQ5 Do the proposed models deliver meaningful generalization bounds without dependence on hypothesis space complexity?

Key findings

• The worst-case expected loss over a Wasserstein ball provides an upper confidence bound on test loss under minimal assumptions.
• For common losses (Huber, epsilon-insensitive, pinball, hinge, logistic) and linear hypotheses, the robust problem reduces to tractable convex programs or kernelizable formulations.
• The approach yields new generalization bounds that can be dimension-independent under bounded hypothesis spaces and concentration results.
• Regularization terms in classical models appear as limits of transportation costs in the Wasserstein framework, offering probabilistic interpretations.
• Robustness under Wasserstein ambiguity aligns with robustness to data perturbations and can recover classical regularization as a special case when transportation cost in the output space grows large.
• The framework supports constructive computation of worst-case distributions for stress testing and calibration.