[Paper Review] Distributionally Robust Logistic Regression

Soroosh Shafieezadeh-Abadeh, Peyman Mohajerin Esfahani+1 more|arXiv (Cornell University)|Sep 30, 2015

Risk and Portfolio Optimization33 references178 citations

TL;DR

This paper proposes a distributionally robust logistic regression model using Wasserstein distance to construct an ambiguity set around the empirical distribution of training data, ensuring out-of-sample performance guarantees. The approach reformulates the robust optimization problem into a tractable conic program that generalizes classical and regularized logistic regression, while providing computable confidence bounds on misclassification risk via linear programming.

ABSTRACT

This paper proposes a distributionally robust approach to logistic regression. We use the Wasserstein distance to construct a ball in the space of probability distributions centered at the uniform distribution on the training samples. If the radius of this ball is chosen judiciously, we can guarantee that it contains the unknown data-generating distribution with high confidence. We then formulate a distributionally robust logistic regression model that minimizes a worst-case expected logloss function, where the worst case is taken over all distributions in the Wasserstein ball. We prove that this optimization problem admits a tractable reformulation and encapsulates the classical as well as the popular regularized logistic regression problems as special cases. We further propose a distributionally robust approach based on Wasserstein balls to compute upper and lower confidence bounds on the misclassification probability of the resulting classifier. These bounds are given by the optimal values of two highly tractable linear programs. We validate our theoretical out-of-sample guarantees through simulated and empirical experiments.

Motivation & Objective

• To address poor out-of-sample performance in classical logistic regression when training data is sparse or noisy.
• To develop a principled regularization framework grounded in distributionally robust optimization, avoiding ad hoc techniques.
• To provide probabilistically interpretable regularization with theoretical guarantees on generalization performance.
• To compute computable upper and lower confidence bounds on the misclassification probability of the resulting classifier.

Proposed method

• Constructs a Wasserstein ball centered at the empirical distribution of training samples, with radius chosen to probabilistically contain the true data-generating distribution.
• Formulates a distributionally robust optimization problem that minimizes the worst-case expected logloss over all distributions in the Wasserstein ball.
• Derives a tractable reformulation of the robust problem using duality theory, resulting in a conic optimization problem involving dual norms and indicator function representations.
• Represents the classification loss as a finite maximum of concave functions to enable convex relaxation and duality application.
• Reformulates the robust problem into a linear program with additional variables and constraints involving dual norms and slack variables.
• Derives computable confidence bounds on misclassification probability as optimal values of two highly tractable linear programs.

Experimental results

Research questions

• RQ1 Can a distributionally robust optimization framework improve the out-of-sample performance of logistic regression under data sparsity or distributional shift?
• RQ2 How can regularization in logistic regression be given a probabilistic interpretation via ambiguity sets in the space of probability distributions?
• RQ3 What is the computational complexity of the resulting robust optimization problem, and can it be reformulated into a tractable form?
• RQ4 Can the proposed method provide rigorous, computable confidence bounds on the classifier's misclassification probability?
• RQ5 How does the proposed method relate to and generalize classical and regularized logistic regression formulations?

Key findings

• The distributionally robust logistic regression problem admits a tractable conic reformulation that generalizes both classical and regularized logistic regression as special cases.
• The robust optimization problem can be solved efficiently via a conic program involving dual norms and slack variables, enabling practical implementation.
• The worst-case risk (misclassification probability) is bounded above and below by the optimal values of two linear programs, providing computable confidence intervals.
• The method provides non-asymptotic, high-probability guarantees that the true data-generating distribution lies within the Wasserstein ball with user-specified confidence.
• The approach naturally incorporates regularization through the Wasserstein radius and dual norm structure, offering a principled alternative to ad hoc regularization.