[Paper Review] Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks

Like Hui, Mikhail Belkin|arXiv (Cornell University)|Jun 12, 2020

Domain Adaptation and Few-Shot Learning42 references53 citations

TL;DR

The paper empirically compares square loss vs cross-entropy across NLP, ASR, and vision tasks, showing square loss often matches or outperforms cross-entropy, especially in NLP/ASR.

ABSTRACT

Modern neural architectures for classification tasks are trained using the cross-entropy loss, which is widely believed to be empirically superior to the square loss. In this work we provide evidence indicating that this belief may not be well-founded. We explore several major neural architectures and a range of standard benchmark datasets for NLP, automatic speech recognition (ASR) and computer vision tasks to show that these architectures, with the same hyper-parameter settings as reported in the literature, perform comparably or better when trained with the square loss, even after equalizing computational resources. Indeed, we observe that the square loss produces better results in the dominant majority of NLP and ASR experiments. Cross-entropy appears to have a slight edge on computer vision tasks. We argue that there is little compelling empirical or theoretical evidence indicating a clear-cut advantage to the cross-entropy loss. Indeed, in our experiments, performance on nearly all non-vision tasks can be improved, sometimes significantly, by switching to the square loss. Furthermore, training with square loss appears to be less sensitive to the randomness in initialization. We posit that training using the square loss for classification needs to be a part of best practices of modern deep learning on equal footing with cross-entropy.

Motivation & Objective

• Assess whether square loss can match or exceed cross-entropy for modern neural classifiers across NLP, ASR, and computer vision tasks.
• Evaluate a range of architectures (CNNs, LSTMs, Transformers) using literature-reported hyper-parameters, with learning-rate adjustments as needed.
• Determine how performance and training stability with square loss compare to cross-entropy under equalized computation budgets.

Proposed method

• Compare square loss and cross-entropy on 28 neural model/dataset combinations across NLP, ASR, and vision domains.
• Use architectures and hyper-parameters from literature for cross-entropy; replace only the loss with square loss (and adjust learning rate).
• Apply loss rescaling for large class counts and remove softmax when training with square loss as recommended.
• Evaluate with domain-specific metrics (accuracy, F1, PER/CER/WER, Top-5 accuracy) over multiple random initializations.
• Run two protocols for square loss: standard training and training with the same number of epochs as CE to equalize compute.

Experimental results

Research questions

• RQ1 Does square loss provide comparable or superior accuracy to cross-entropy across diverse architectures and datasets?
• RQ2 How does the square loss perform relative to cross-entropy when computational resources are equalized (same epochs) across tasks?
• RQ3 Are there domain-dependent patterns in loss-function performance (e.g., NLP/ASR vs. computer vision)?

Key findings

• Square loss yields better or equal accuracy than cross-entropy in 22 of 28 tasks overall.
• Square loss generally shows smaller variance across random initializations in most experiments.
• For NLP and ASR, square loss often outperforms cross-entropy; for vision, cross-entropy has a slight edge overall, with exceptions (e.g., EfficientNet on ImageNet).
• Equalizing computation by matching epochs does not remove the square loss advantage in non-vision tasks and can preserve or improve performance.
• Removal of the softmax layer is recommended when training with square loss to avoid optimization hindrance; loss rescaling helps with many-classes problems.