[论文解读] Post-hoc Calibration of Neural Networks.

Amir Rahimi, Kartik Gupta+4 more|arXiv (Cornell University)|Jun 23, 2020

Adversarial Robustness in Machine Learning参考文献 22被引用 4

一句话总结

本文通过证明即使基础网络并非最优，仅通过在基础网络（f）的输出上最小化负对数似然（NLL）并添加一个可学习的校准头（g），即可实现最终模型（g∘f）的校准，为神经网络的后训练校准提供了理论基础。在图像分类基准上的实验验证了理论发现。

ABSTRACT

Calibration of neural networks is a critical aspect to consider when incorporating machine learning models in real-world decision-making systems where the confidence of decisions are equally important as the decisions themselves. In recent years, there is a surge of research on neural network calibration and the majority of the works can be categorized into post-hoc calibration methods, defined as methods that learn an additional function to calibrate an already trained base network. In this work, we intend to understand the post-hoc calibration methods from a theoretical point of view. Especially, it is known that minimizing Negative Log-Likelihood (NLL) will lead to a calibrated network on the training set if the global optimum is attained (Bishop, 1994). Nevertheless, it is not clear learning an additional function in a post-hoc manner would lead to calibration in the theoretical sense. To this end, we prove that even though the base network ($f$) does not lead to the global optimum of NLL, by adding additional layers ($g$) and minimizing NLL by optimizing the parameters of $g$ one can obtain a calibrated network $g \circ f$. This not only provides a less stringent condition to obtain a calibrated network but also provides a theoretical justification of post-hoc calibration methods. Our experiments on various image classification benchmarks confirm the theory.

研究动机与目标

• 理解神经网络后训练校准方法的理论基础。
• 研究在训练基础网络后学习附加函数是否能实现理论上的校准。
• 放宽对基础网络必须达到NLL全局最优的限制，以实现校准。
• 为后训练校准技术的实证成功提供理论依据。

提出的方法

• 本文引入两阶段模型：预训练的基础网络f和可学习的校准头部g，构成复合模型g∘f。
• 证明通过优化g的参数来最小化f输出的NLL，即使f未达到全局最优，也能得到校准的模型。
• 理论分析依赖于一个已知性质：当达到全局最优时，最小化NLL可使训练集实现校准。
• 该方法将基础网络f视为固定，仅优化g的参数以校准输出。
• 该方法适用于任何基础网络和校准头部架构，只要g能够建模必要的变换。

实验结果

研究问题

• RQ1 即使基础网络f未达到全局最优，通过可学习头部g进行后训练校准是否能实现理论校准？
• RQ2 在优化g的参数后，什么条件能确保复合模型g∘f是校准的？
• RQ3 通过g在f的输出上最小化NLL，如何与最终模型的校准相关联？
• RQ4 该理论依据在多种图像分类基准上是否在实践中成立？

主要发现

• 理论分析证明，即使基础网络f未达到全局最优，通过可学习头部g在预训练基础网络f的输出上最小化NLL，也能得到校准的最终模型g∘f。
• 该方法对基础网络达到NLL全局最优的要求更为宽松，从而降低了校准的门槛。
• 理论依据支持了后训练校准技术在实践中取得成功的原因。
• 在多个图像分类基准上的实验验证了理论主张，表明该方法能有效实现校准。