[Paper Review] Rethinking Softmax Cross-Entropy Loss for Adversarial Robustness

Tianyu Pang, Kun Xu+4 more|arXiv (Cornell University)|May 25, 2019

Adversarial Robustness in Machine LearningComputer Science72 references75 citations

TL;DR

The paper introduces the Max-Mahalanobis Center (MMC) loss to explicitly induce dense, well-structured feature regions, improving adversarial robustness while preserving clean accuracy and offering faster convergence compared to Softmax Cross-Entropy (SCE)—even under strong adaptive attacks.

ABSTRACT

Previous work shows that adversarially robust generalization requires larger sample complexity, and the same dataset, e.g., CIFAR-10, which enables good standard accuracy may not suffice to train robust models. Since collecting new training data could be costly, we focus on better utilizing the given data by inducing the regions with high sample density in the feature space, which could lead to locally sufficient samples for robust learning. We first formally show that the softmax cross-entropy (SCE) loss and its variants convey inappropriate supervisory signals, which encourage the learned feature points to spread over the space sparsely in training. This inspires us to propose the Max-Mahalanobis center (MMC) loss to explicitly induce dense feature regions in order to benefit robustness. Namely, the MMC loss encourages the model to concentrate on learning ordered and compact representations, which gather around the preset optimal centers for different classes. We empirically demonstrate that applying the MMC loss can significantly improve robustness even under strong adaptive attacks, while keeping state-of-the-art accuracy on clean inputs with little extra computation compared to the SCE loss.

Motivation & Objective

• Motivate robust classification with limited or fixed training data by shaping the feature distribution instead of relying on more data.
• Formally analyze how SCE and its variants influence feature space density and identify limitations.
• Propose MMC loss to induce high-density, centered feature regions for each class.
• Demonstrate robustness benefits of MMC under various threat models while maintaining clean accuracy and efficient training.

Proposed method

• Define sample density in feature space and analyze density under generalized SCE (g-SCE) losses.
• Show that SCE and variants yield sparse feature distributions due to softmax-based supervision.
• Propose MMC loss: L_MMC(Z(x),y) = 1/2 || z - μ_y^* ||^2, with fixed centers μ^* forming a Max-Mahalanobis Distribution (MMD) for optimal inter-class dispersion.
• Compute and leverage pre-set class centers μ^* that maximize minimal angles between centers (||μ_l^*||_2 = C_MM).
• Argue that MMC yields higher local feature density (Theorem 2) and better exploitation of model capacity without degradation problems.
• Compare MMC to Center, MMLDA, and L-GM losses across MNIST, CIFAR-10, and CIFAR-100 with and without adversarial training.

Experimental results

Research questions

• RQ1 How do SCE-based losses shape the density of learned features in the feature space?
• RQ2 Can explicitly inducing high-density, centered feature regions improve adversarial robustness without sacrificing clean accuracy?
• RQ3 What is the impact of using fixed class centers (μ^*) on robustness and convergence compared to trainable centers?
• RQ4 How does MMC perform under adaptive white-box and black-box adversarial attacks relative to standard losses and prior center-based methods?

Key findings

• MMC yields higher local feature density around fixed class centers, promoting robustness with limited extra computation.
• MMC achieves robustness under strong adaptive attacks while maintaining clean accuracy comparable to SCE baselines.
• A center set sampled randomly (MMC-10 rand) already outperforms several baselines, with optimal μ^* providing further gains.
• MMC converges faster and better utilizes model capacity than center-based losses that require co-optimization with SCE.
• When combined with adversarial training, MMC improves robustness under attacks different from the training attack.
• MMC avoids degradation issues because centers are fixed and do not drift during training.