[Paper Review] Consistent Robust Adversarial Prediction for General Multiclass Classification

Rizal Fathony, Kaiser Asif+6 more|arXiv (Cornell University)|Dec 18, 2018

Adversarial Robustness in Machine Learning45 references4 citations

TL;DR

This paper proposes a robust adversarial prediction framework for general multiclass classification that formulates the optimization of non-convex, non-continuous multiclass loss metrics as a convex dual problem via an adversarial surrogate loss. The method guarantees Fisher consistency, enables kernel-based feature representation, and achieves competitive performance across multiple datasets, outperforming baselines in both linear and nonlinear settings.

ABSTRACT

We propose a robust adversarial prediction framework for general multiclass classification. Our method seeks predictive distributions that robustly optimize non-convex and non-continuous multiclass loss metrics against the worst-case conditional label distributions (the adversarial distributions) that (approximately) match the statistics of the training data. Although the optimized loss metrics are non-convex and non-continuous, the dual formulation of the framework is a convex optimization problem that can be recast as a risk minimization model with a prescribed convex surrogate loss we call the adversarial surrogate loss. We show that the adversarial surrogate losses fill an existing gap in surrogate loss construction for general multiclass classification problems, by simultaneously aligning better with the original multiclass loss, guaranteeing Fisher consistency, enabling a way to incorporate rich feature spaces via the kernel trick, and providing competitive performance in practice.

Motivation & Objective

• To address the lack of theoretically sound, consistent surrogate losses for general multiclass classification, especially for non-convex and non-continuous loss metrics like zero-one and ordinal losses.
• To develop a robust prediction framework that optimizes against the worst-case conditional label distributions matching training data statistics via moment-matching constraints.
• To construct a convex surrogate loss—called the adversarial surrogate loss—that preserves the original loss structure while ensuring Fisher consistency and computational efficiency.
• To enable the use of rich feature spaces through the kernel trick in multiclass settings where such extensions are typically problematic.
• To empirically validate the method's superiority over existing baselines in terms of abstention loss and classification accuracy across diverse datasets.

Proposed method

• The framework formulates multiclass prediction as a robust optimization problem over adversarial label distributions that approximately match training data moments, ensuring distributional robustness.
• It derives a dual formulation of the adversarial problem, which reduces to a convex empirical risk minimization problem with a novel convex surrogate loss—termed the adversarial surrogate loss.
• The adversarial surrogate loss is constructed to align closely with the original multiclass loss metric, guaranteeing Fisher consistency and enabling kernel-based feature mapping.
• Efficient algorithms are developed: linear time for ordinal classification with absolute loss, quasilinear time for zero-one loss, and linear programming for general metrics.
• The method supports classification with abstention by learning a rejection threshold, allowing models to avoid uncertain predictions.
• The framework is extended to nonlinear settings via the kernel trick, maintaining computational efficiency and theoretical guarantees.

Experimental results

Research questions

• RQ1 Can a robust adversarial prediction framework be designed to optimize non-convex and non-continuous multiclass loss metrics while ensuring theoretical consistency?
• RQ2 Does the proposed adversarial surrogate loss achieve Fisher consistency for multiclass classification, particularly for zero-one and ordinal loss metrics?
• RQ3 Can the adversarial surrogate loss be efficiently optimized and extended to rich feature spaces using the kernel trick?
• RQ4 How does the method compare to existing baselines in terms of classification accuracy and abstention performance across diverse multiclass datasets?
• RQ5 Does the inclusion of abstention capability improve robustness and performance in noisy or complex classification tasks?

Key findings

• The adversarial surrogate loss guarantees Fisher consistency for multiclass classification, filling a long-standing theoretical gap in surrogate loss construction.
• The method achieves competitive performance across multiple UCI datasets, with AL^abstain outperforming SVM-OVA and SVM-CS in terms of overall abstention loss and number of statistically indistinguishable best results.
• In linear kernel experiments, AL^abstain achieved the best or near-best performance on most datasets, with a slight edge over SVM-CS in several cases.
• The introduction of the Gaussian kernel significantly improved performance for both AL^abstain and SVM-CS, with AL^abstain maintaining its advantage across all datasets.
• Models using the proposed framework produced more abstain predictions in higher-noise datasets (e.g., redwine), indicating sensitivity to uncertainty and improved reliability in challenging settings.
• The computational cost of the adversarial surrogate loss remains low and comparable to baselines, with efficient algorithms enabling linear or quasilinear time complexity for key cases.