[Paper Review] Evaluating the Impact of Loss Function Variation in Deep Learning for Classification

Simon Dräger, Jannik Dunkelau|arXiv (Cornell University)|Oct 28, 2022

Machine Learning and Data Classification4 citations

TL;DR

This paper empirically evaluates the impact of different loss functions—MSE, MAE, CCE, and KLD—on deep learning classification performance across multiple datasets. It finds that the Kullback-Leibler Divergence (KLD) outperforms the widely used Cross-Entropy (CCE) loss on four out of six datasets, challenging the dogmatic adoption of CCE and advocating for loss functions to be treated as tunable hyperparameters rather than fixed choices.

ABSTRACT

The loss function is arguably among the most important hyperparameters for a neural network. Many loss functions have been designed to date, making a correct choice nontrivial. However, elaborate justifications regarding the choice of the loss function are not made in related work. This is, as we see it, an indication of a dogmatic mindset in the deep learning community which lacks empirical foundation. In this work, we consider deep neural networks in a supervised classification setting and analyze the impact the choice of loss function has onto the training result. While certain loss functions perform suboptimally, our work empirically shows that under-represented losses such as the KL Divergence can outperform the State-of-the-Art choices significantly, highlighting the need to include the loss function as a tuned hyperparameter rather than a fixed choice.

Motivation & Objective

• To challenge the widespread, unexamined adoption of Cross-Entropy (CCE) as the default loss function in deep learning classification.
• To empirically assess how varying loss functions affects model performance, convergence, and stability across diverse datasets.
• To demonstrate that underutilized losses like KLD can significantly outperform state-of-the-art choices, especially in multi-class settings.
• To advocate for treating the loss function as a hyperparameter to be optimized, not a fixed axiom.
• To encourage further research into custom and information-theoretic loss functions, such as f-divergences.

Proposed method

• The study evaluates four standard loss functions: Mean Squared Error (MSE), Mean Absolute Error (MAE), Categorical Cross-Entropy (CCE), and Kullback-Leibler Divergence (KLD).
• Each loss function is applied to the same deep neural network architecture across six benchmark datasets: Mushroom, Phishing, MNIST, Fashion-MNIST, CIFAR-10, and CharFontIm.
• Models are trained for 100 epochs using the same optimizer (Adam) and hyperparameters, with performance measured via test accuracy, standard error of the mean (SEM), F1-score (ϕ), and AUC.
• The KLD is mathematically decomposed as CCE minus label entropy, highlighting its information-theoretic interpretation as the excess bits needed to encode data.
• Experiments are repeated across multiple random seeds to compute mean performance and standard error, ensuring statistical reliability.
• Performance is compared across loss functions using both accuracy and robustness metrics (ϕ and AUC), with results reported in tables for direct comparison.

Experimental results

Research questions

• RQ1 Does the choice of loss function significantly impact the final classification accuracy of deep neural networks?
• RQ2 Can less common loss functions like KLD outperform the widely adopted Cross-Entropy (CCE) loss in standard classification tasks?
• RQ3 How do different loss functions affect model stability and convergence, as measured by standard error and robustness metrics?
• RQ4 Why does MAE perform poorly on complex image datasets despite its robustness to outliers in regression?
• RQ5 To what extent should loss functions be treated as hyperparameters rather than fixed defaults in deep learning pipelines?

Key findings

• On the MNIST dataset, MAE achieved a test accuracy of 99.30% ± 0.01%, outperforming CCE (98.99% ± 0.03%) and KLD (98.84% ± 0.04%), suggesting it may be effective for learning conditional medians.
• On the Mushroom dataset, KLD achieved 100.00% ± 0.00% accuracy, significantly outperforming CCE (99.67% ± 0.10%) and MSE (99.69% ± 0.26%), with perfect F1-score and AUC.
• On Fashion-MNIST, KLD achieved 90.47% ± 0.11% accuracy, surpassing CCE (88.60% ± 0.11%) and MSE (87.96% ± 0.04%), with the highest F1-score (0.89) and AUC (0.99).
• On CIFAR-10, CCE achieved 78.13% ± 0.28% accuracy, the highest among all losses, while KLD achieved 70.05% ± 0.30%, indicating that CCE remains superior for complex color image classification.
• On CharFontIm, KLD achieved 76.62% ± 0.02% accuracy and AUC of 0.99, outperforming CCE (69.21% ± 0.97% accuracy) and MAE (3.37% ± 1.52%), demonstrating strong performance on non-standard image data.
• The KLD consistently achieved the best or near-best performance on four out of six datasets, with high F1-scores and AUC values, suggesting strong generalization and robustness in multi-class settings.