[Paper Review] Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning

Elliott Gordon-Rodríguez, Gabriel Loaiza-Ganem+2 more|arXiv (Cornell University)|Nov 10, 2020

Adversarial Robustness in Machine LearningComputer Science28 references55 citations

TL;DR

The paper critiques using cross-entropy on simplex-valued targets and proposes probabilistic alternatives based on the continuous-categorical distribution for label smoothing (CC-LS) and actor-mimic reinforcement learning (CC-AMN), with mixed empirical results across tasks.

ABSTRACT

Modern deep learning is primarily an experimental science, in which empirical advances occasionally come at the expense of probabilistic rigor. Here we focus on one such example; namely the use of the categorical cross-entropy loss to model data that is not strictly categorical, but rather takes values on the simplex. This practice is standard in neural network architectures with label smoothing and actor-mimic reinforcement learning, amongst others. Drawing on the recently discovered continuous-categorical distribution, we propose probabilistically-inspired alternatives to these models, providing an approach that is more principled and theoretically appealing. Through careful experimentation, including an ablation study, we identify the potential for outperformance in these models, thereby highlighting the importance of a proper probabilistic treatment, as well as illustrating some of the failure modes thereof.

Motivation & Objective

• Motivate the need for a probabilistic treatment when targets lie on the simplex rather than being strictly categorical.
• Introduce continuous-categorical (CC) log-likelihood as a principled replacement for cross-entropy on simplex-valued data.
• Evaluate CC-LS as a replacement for label smoothing in CIFAR-10 to assess regularization and representation learning.
• Evaluate CC-AMN as a replacement for cross-entropy in actor-mimic reinforcement learning on Atari games.
• Discuss implications and limitations of CC-based likelihoods for deep learning practice.

Proposed method

• Formulate the continuous-categorical (CC) distribution as a normalized version of the cross-entropy likelihood with a closed-form normalizing constant C(λ) given by a specific log-ratio expression.
• Replace the cross-entropy loss with the CC log-likelihood: l(λ;y) = -log C(λ) - sum_k y_k log λ_k (Equation 2).
• Apply CC to label smoothing by using simplex-valued targets y^LS in place of one-hot labels, defining CC-LS with the CC objective (Equation 7).
• Apply CC to actor-mimic reinforcement learning by recasting the AMN training objective as CC-AMN (Equation 11).
• Conduct ablation studies and ablations on regularizers to compare LS, CC-LS, and baseline under varying BatchNorm, dropout, and weight decay settings.
• Evaluate numerical stability and scalability limitations of the CC normalizing constant, especially for larger K.

Experimental results

Research questions

• RQ1 Does replacing cross-entropy with CC log-likelihood improve regularization or representation learning in label smoothing?
• RQ2 Can CC-LS outperform vanilla label smoothing across common CNN regularization regimes?
• RQ3 Can CC-AMN offer advantages over standard AMN in multi-task reinforcement learning settings, and what are its failure modes?

Key findings

• CC-LS significantly outperforms LS and baseline in an unregularized CNN on CIFAR-10, while BatchNorm can negate gains from both LS and CC-LS.
• CC-LS provides a distinct regularization effect that can yield richer learned representations than vanilla LS, particularly when batch normalization is absent.
• CC-AMN achieves performance similar to AMN on most Atari games, but shows instability and worse results on Pong due to numerical issues in the normalizing constant near uniform λ.
• In ablations, CC-LS shows gains in absence of BatchNorm and can reduce test accuracy variability under weight decay without BatchNorm.
• CC-LS and CC-AMN share the insight that a probabilistic interpretation of simplex-valued targets can yield benefits, though architecture and initialization effects remain important.
• CC-AMN exhibits a notable failure mode tied to numerical instability when λ is near uniform, highlighting practical challenges for high-dimensional simplex targets.