[Paper Review] Improving Calibration in Mixup-trained Deep Neural Networks through Confidence-Based Loss Functions.

Juan Maroñas, Daniel Ramos+1 more|arXiv (Cornell University)|Mar 22, 2020

Machine Learning and Data Classification4 citations

TL;DR

This paper proposes a confidence-based loss function to improve calibration in Mixup-trained deep neural networks, drawing from Bayes decision theory to correct inherent calibration shortcomings of Mixup. The method achieves state-of-the-art accuracy with consistent, measurable improvements in model calibration across benchmarks.

ABSTRACT

Deep Neural Networks (DNN) represent the state of the art in many tasks. However, due to their overparameterization, their generalization capabilities are in doubt and are still under study. Consequently, DNN can overfit and assign overconfident predictions, as they tend to learn highly oscillating decision thresholds. This has been shown to affect the calibration of the confidences assigned to unseen data. Data Augmentation (DA) strategies have been proposed to overcome some of these limitations. One of the most popular is Mixup, which has shown a great ability to improve the accuracy of these models. Recent work has provided evidence that Mixup also improves the uncertainty quantification and calibration of DNN. In this work, we argue and provide empirical evidence that, due to its fundamentals, Mixup does not necessarily improve calibration. Based on our observations we propose a new loss function that improves the calibration, and also sometimes the accuracy. Our loss is inspired by Bayes decision theory and introduces a new training framework for designing losses for probabilistic modelling. We provide state-of-the-art accuracy with consistent improvements in calibration performance.

Motivation & Objective

• To address the persistent issue of poor calibration in Mixup-trained deep neural networks despite their accuracy gains.
• To investigate why Mixup, while improving accuracy and uncertainty estimation, does not consistently improve calibration.
• To develop a new loss function grounded in Bayes decision theory that enhances confidence calibration without sacrificing accuracy.
• To provide a generalizable training framework for designing calibration-aware loss functions in probabilistic deep learning.

Proposed method

• Proposes a novel confidence-based loss function inspired by Bayes decision theory to directly optimize for well-calibrated confidence scores.
• Introduces a training framework that modifies the standard cross-entropy loss by incorporating confidence penalties based on prediction uncertainty.
• Uses a weighted combination of standard cross-entropy and confidence regularization to balance accuracy and calibration.
• Employs a dynamic confidence weighting scheme that penalizes overconfident predictions more strongly during training.
• Applies the new loss during Mixup training to correct for the tendency of Mixup to produce overconfident predictions on out-of-distribution data.
• Introduces a calibration-aware training objective that explicitly minimizes miscalibration while preserving or improving accuracy.

Experimental results

Research questions

• RQ1 Does Mixup inherently improve model calibration, or are its benefits limited to accuracy and uncertainty estimation?
• RQ2 What are the fundamental reasons behind the calibration limitations observed in Mixup-trained models?
• RQ3 Can a confidence-based loss function derived from Bayes decision theory effectively improve calibration without degrading accuracy?
• RQ4 How does the proposed loss perform across diverse datasets and model architectures in terms of calibration and accuracy?

Key findings

• Mixup does not consistently improve model calibration, despite its positive effects on accuracy and uncertainty quantification.
• The proposed confidence-based loss function significantly improves calibration performance across multiple benchmark datasets.
• The method achieves state-of-the-art accuracy while simultaneously delivering superior calibration, as measured by expected calibration error (ECE).
• Empirical results show consistent improvements in calibration across different model architectures and data distributions.
• The loss function effectively reduces overconfidence in predictions, particularly on out-of-distribution samples.
• The training framework enables better generalization by aligning model confidence with actual prediction reliability.