[Paper Review] Understanding Square Loss in Training Overparametrized Neural Network Classifiers

Tianyang Hu, Jun Wang+2 more|arXiv (Cornell University)|Dec 7, 2021

Advanced Neural Network Applications4 citations

TL;DR

This paper provides a theoretical and empirical analysis of square loss in training overparametrized neural network classifiers, showing it achieves fast convergence in generalization and calibration error, exponential convergence in separable cases, and superior robustness. It demonstrates that square loss with simplex coding yields better calibration and adversarial robustness than cross-entropy, especially under noise and adversarial attacks.

ABSTRACT

Deep learning has achieved many breakthroughs in modern classification tasks. Numerous architectures have been proposed for different data structures but when it comes to the loss function, the cross-entropy loss is the predominant choice. Recently, several alternative losses have seen revived interests for deep classifiers. In particular, empirical evidence seems to promote square loss but a theoretical justification is still lacking. In this work, we contribute to the theoretical understanding of square loss in classification by systematically investigating how it performs for overparametrized neural networks in the neural tangent kernel (NTK) regime. Interesting properties regarding the generalization error, robustness, and calibration error are revealed. We consider two cases, according to whether classes are separable or not. In the general non-separable case, fast convergence rate is established for both misclassification rate and calibration error. When classes are separable, the misclassification rate improves to be exponentially fast. Further, the resulting margin is proven to be lower bounded away from zero, providing theoretical guarantees for robustness. We expect our findings to hold beyond the NTK regime and translate to practical settings. To this end, we conduct extensive empirical studies on practical neural networks, demonstrating the effectiveness of square loss in both synthetic low-dimensional data and real image data. Comparing to cross-entropy, square loss has comparable generalization error but noticeable advantages in robustness and model calibration.

Motivation & Objective

• To provide a theoretical justification for the empirical success of square loss in deep learning classifiers, especially in overparametrized settings.
• To investigate how square loss affects generalization error, calibration error, and robustness in both separable and non-separable classification settings.
• To compare square loss with cross-entropy in terms of model calibration, adversarial robustness, and performance under Gaussian noise.
• To evaluate the impact of label coding schemes—specifically one-hot vs. simplex coding—on square loss performance.
• To develop and validate a novel angle attack tailored for square loss classifiers to assess their robustness more accurately.

Proposed method

• Analyzes square loss in the neural tangent kernel (NTK) regime to derive theoretical convergence rates for misclassification and calibration errors.
• Derives the optimal classifier under square loss as $2\eta - 1$, which is linear in the true conditional probability $\eta$, enabling better calibration.
• Proposes an angle attack that maximizes cosine similarity instead of square loss, revealing stronger adversarial vulnerability in square loss models.
• Employs both white-box (PGD) and black-box (AutoAttack) adversarial attacks to evaluate robustness on CIFAR-10 and CIFAR-100.
• Evaluates robustness to Gaussian noise by injecting noise with standard deviations from 0.1 to 0.4 and measuring accuracy drop.
• Compares one-hot and simplex label coding schemes, with simplex coding shown to improve performance on deeper networks like WRN-16-10.

Experimental results

Research questions

• RQ1 How does square loss perform in terms of generalization and calibration error in the overparametrized NTK regime?
• RQ2 What are the theoretical convergence rates of misclassification and calibration error under square loss for separable and non-separable data?
• RQ3 How does square loss compare to cross-entropy in adversarial robustness, especially under PGD and AutoAttack?
• RQ4 What is the impact of label coding (one-hot vs. simplex) on the performance of square loss classifiers?
• RQ5 Can a specialized attack—angle attack—reveal stronger vulnerabilities in square loss models than standard square loss maximization?

Key findings

• In the non-separable case, square loss achieves a fast convergence rate for both misclassification rate and calibration error in the NTK regime.
• When classes are separable, the misclassification rate under square loss converges exponentially fast, and the margin is lower bounded away from zero, ensuring robustness.
• Square loss-trained models show comparable generalization error to cross-entropy but significantly better calibration and robustness under adversarial attacks.
• On CIFAR-10 and CIFAR-100, square loss models maintain higher accuracy under Gaussian noise (e.g., 90.07% at σ=0.1 on ResNet-18 vs. 90.03% for cross-entropy), indicating superior noise robustness.
• The angle attack outperforms standard PGD attacks on square loss models, revealing that standard attacks may underestimate their vulnerability.
• Simplex coding consistently outperforms scaled one-hot coding, especially on deeper networks like WRN-16-10, where it achieves 79.65% test accuracy vs. 78.39% for one-hot coding.