[Paper Review] Robustified Multivariate Regression and Classification Using Distributionally Robust Optimization under the Wasserstein Metric

Ruidi Chen, Ioannis Ch. Paschalidis|arXiv (Cornell University)|Jun 10, 2020

Advanced Statistical Methods and Models19 references4 citations

TL;DR

This paper proposes a distributionally robust optimization framework for multivariate regression and classification using the Wasserstein metric to handle outliers in both covariates and responses. By reformulating the min-max Wasserstein DRO problem, it derives a regularized learning model with a matrix norm regularizer that links robustness to regularization, achieving 7–37% lower predictive error in multivariate regression and 100% higher robustness (measured by minimal perturbation distance) in multiclass logistic regression compared to baselines.

ABSTRACT

We develop Distributionally Robust Optimization (DRO) formulations for Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG) when both the covariates and responses/labels may be contaminated by outliers. The DRO framework uses a probabilistic ambiguity set defined as a ball of distributions that are close to the empirical distribution of the training set in the sense of the Wasserstein metric. We relax the DRO formulation into a regularized learning problem whose regularizer is a norm of the coefficient matrix. We establish out-of-sample performance guarantees for the solutions to our model, offering insights on the role of the regularizer in controlling the prediction error. Experimental results show that our approach improves the predictive error by 7% -- 37% for MLR, and a metric of robustness by 100% for MLG.

Motivation & Objective

• To develop a robust multivariate learning framework that is resilient to outliers in both covariates and responses.
• To extend distributionally robust optimization (DRO) from univariate to multivariate settings, specifically for Multivariate Linear Regression (MLR) and Multiclass Logistic Regression (MLG).
• To establish a theoretical and computational connection between robustness and regularization in the multivariate case via the Wasserstein metric.
• To provide out-of-sample performance guarantees for the proposed robust estimators under distributional shifts and contamination.
• To empirically demonstrate superior predictive accuracy and robustness compared to standard and regularized baselines on datasets with covariate shift and outliers.

Proposed method

• Formulates a min-max distributionally robust optimization problem minimizing the worst-case expected loss over a Wasserstein ambiguity set centered on the empirical distribution of training data.
• Reformulates the Wasserstein DRO problem into a regularized empirical loss minimization problem by leveraging duality, resulting in a matrix norm regularizer derived from the dual norm of the data distance metric.
• Derives a novel matrix-norm regularizer for the coefficient matrix in MLR and MLG that captures geometric structure and correlations among multiple responses.
• Applies the relaxation to both MLR with Lipschitz loss and MLG with log-loss, enabling tractable optimization and primal-dual interpretation of data-coefficient relationships.
• Uses the Wasserstein metric to define a probabilistic ambiguity set that captures distributional shifts and contamination, particularly effective under covariate shift.
• Employs a systematic integration of multivariate responses rather than independent univariate modeling, avoiding simple superposition and preserving response correlations.

Experimental results

Research questions

• RQ1 How can distributionally robust optimization be extended to multivariate regression and classification problems with multiple correlated responses?
• RQ2 What is the structure of the robust counterpart for multivariate learning under the Wasserstein metric, and how does it differ from univariate DRO formulations?
• RQ3 How does the resulting regularizer in the relaxed problem relate to robustness and what is its geometric interpretation in the coefficient matrix space?
• RQ4 Can the proposed framework achieve better out-of-sample performance and robustness under data contamination and covariate shift compared to standard and regularized baselines?
• RQ5 What is the empirical impact of the proposed method on predictive accuracy and adversarial robustness, especially in terms of minimal perturbation distance?

Key findings

• The proposed MLR model reduces predictive error by 7% to 37% compared to standard multivariate regression under outlier contamination.
• For multiclass logistic regression, the MLG-SR and MLG-1S models achieve a 100% improvement in minimal perturbation distance (MPD), indicating significantly higher robustness to input perturbations.
• The MLG-1S model improves over PCC MLG by 12% in Correct Classification Rate (CCR), 13% in log-loss, and 16% in CVaR, despite not explicitly modeling predictor correlations.
• The method provides theoretical out-of-sample performance guarantees by linking the regularizer to the worst-case risk under distributional shifts.
• The matrix-norm regularizer derived from the Wasserstein DRO framework effectively captures multivariate dependencies and enables efficient optimization without modeling response correlation explicitly.
• Empirical results confirm that the optimization-based approach outperforms statistical methods that explicitly model response or predictor correlations, especially under covariate shift and data contamination.