[Paper Review] Can multi-label classification networks know what they don't know?

Haoran Wang, Weitang Liu+2 more|arXiv (Cornell University)|Sep 29, 2021

Anomaly Detection Techniques and ApplicationsComputer Science50 references35 citations

TL;DR

JointEnergy aggregates label-wise energy across all labels to detect out-of-distribution in multi-label classification, achieving state-of-the-art OOD detection and a joint-likelihood interpretation.

ABSTRACT

Estimating out-of-distribution (OOD) uncertainty is a central challenge for safely deploying machine learning models in the open-world environment. Improved methods for OOD detection in multi-class classification have emerged, while OOD detection methods for multi-label classification remain underexplored and use rudimentary techniques. We propose JointEnergy, a simple and effective method, which estimates the OOD indicator scores by aggregating energy scores from multiple labels. We show that JointEnergy can be mathematically interpreted from a joint likelihood perspective. Our results show consistent improvement over previous methods that are based on the maximum-valued scores, which fail to capture joint information from multiple labels. We demonstrate the effectiveness of our method on three common multi-label classification benchmarks, including MS-COCO, PASCAL-VOC, and NUS-WIDE. We show that JointEnergy can reduce the FPR95 by up to 10.05% compared to the previous best baseline, establishing state-of-the-art performance.

Motivation & Objective

• Motivate OOD uncertainty estimation in multi-label classification, where multiple labels can be present simultaneously.
• Develop a scalable method that leverages joint information across labels without training generative models.
• Provide theoretical interpretation of JointEnergy from a joint likelihood perspective.
• Empirically demonstrate state-of-the-art OOD detection on standard multi-label benchmarks.

Proposed method

• Define label-wise free energy for each label as E_yi(x) = -log(1 + exp(f_yi(x))).
• Propose JointEnergy as the sum of negated label-wise energies across all K labels: E_joint(x) = sum_i (-E_yi(x)).
• Interpret JointEnergy theoretically via a joint likelihood perspective, connecting it to p(x | y1=1,...,yK=1) and data density.
• Show that JointEnergy emphasizes joint information across labels, enabling better separation of in-distribution and OOD data than MaxLogit.
• Demonstrate hyperparameter-free usage and compatibility with different label-wise scoring functions and aggregation schemes.

Experimental results

Research questions

• RQ1 How can OOD uncertainty be effectively estimated in multi-label classification settings?
• RQ2 Does aggregating energy scores across all labels outperform using a single dominant label (e.g., MaxLogit) for OOD detection?
• RQ3 What theoretical interpretation supports JointEnergy as an OOD score in multi-label scenarios?
• RQ4 How does JointEnergy perform across common multi-label benchmarks and architectures when detecting OOD data?

Key findings

• JointEnergy achieves state-of-the-art OOD detection on MS-COCO, PASCAL-VOC, and NUS-WIDE benchmarks, reducing FPR95 by up to 10.05% compared to the previous best baseline.
• On MS-COCO, JointEnergy yields FPR95 33.48, AUROC 92.70, and AUPR 96.25, outperforming competing methods.
• On PASCAL-VOC, JointEnergy yields FPR95 41.01, AUROC 91.10, and AUPR 86.33.
• On NUS-WIDE, JointEnergy yields FPR95 48.98, AUROC 88.30, and AUPR 96.40.
• Ablation shows that summing label energies (JointEnergy) outperforms summing other label-wise scores (e.g., logits, MSP, ODIN, Mahalanobis) and MaxEnergy, highlighting the importance of joint information across labels.
• JointEnergy provides a mathematically interpretable joint-likelihood perspective, relating higher scores to higher joint likelihood of all labels being present.