[Paper Review] Towards Maximizing the Representation Gap between In-Domain & Out-of-Distribution Examples

Jay Nandy, Wynne Hsu+1 more|arXiv (Cornell University)|Oct 20, 2020

Domain Adaptation and Few-Shot LearningComputer Science36 references20 citations

TL;DR

This paper proposes a novel loss function for Dirichlet Prior Networks (DPN) that maximizes the representation gap between in-domain and out-of-distribution (OOD) examples by explicitly regularizing the precision of Dirichlet distributions. By decoupling mean and precision optimization, the method improves OOD detection performance, achieving state-of-the-art results across multiple benchmarks, including near-perfect AUROC scores on CIFAR-10 and STL-10.

ABSTRACT

Among existing uncertainty estimation approaches, Dirichlet Prior Network (DPN) distinctly models different predictive uncertainty types. However, for in-domain examples with high data uncertainties among multiple classes, even a DPN model often produces indistinguishable representations from the out-of-distribution (OOD) examples, compromising their OOD detection performance. We address this shortcoming by proposing a novel loss function for DPN to maximize the extit{representation gap} between in-domain and OOD examples. Experimental results demonstrate that our proposed approach consistently improves OOD detection performance.

Motivation & Objective

• To address the limitation in existing DPN models where high data uncertainty leads to flat Dirichlet distributions indistinguishable from OOD examples.
• To improve OOD detection by explicitly maximizing the representation gap between in-domain and OOD examples.
• To design a loss function that separately models the mean and precision of Dirichlet outputs for better uncertainty disentanglement.
• To achieve robust OOD detection performance even in high-data-uncertainty scenarios for in-domain examples.

Proposed method

• Proposes a new loss function that decouples the cross-entropy loss from a novel explicit precision regularizer to separately optimize mean and precision of Dirichlet distributions.
• Introduces a precision regularizer that encourages sharp, multi-modal Dirichlet distributions for OOD examples, increasing their representation gap from in-domain examples.
• Modifies the standard DPN loss to avoid distributing target precision values among overlapping classes in high-uncertainty in-domain cases.
• Uses a two-stage optimization: first, standard cross-entropy loss for mean prediction; second, precision regularizer to enforce sparsity and sharpness in OOD predictions.
• Employs a target precision value that is dynamically adjusted to promote flatness for OOD and sharpness for in-domain examples.
• Trains the model using both in-domain and OOD examples to explicitly learn the distinction in representation space.

Experimental results

Research questions

• RQ1 Can a modified DPN loss function effectively increase the representation gap between in-domain and OOD examples?
• RQ2 Does separating mean and precision optimization in DPN improve OOD detection under high data uncertainty?
• RQ3 Can explicit precision regularization prevent in-domain examples with high uncertainty from being misclassified as OOD?
• RQ4 How does the proposed method compare to existing OOD detection baselines across diverse datasets?
• RQ5 Does the method maintain high performance across different data distributions and OOD shift types?

Key findings

• On CIFAR-10, the proposed DPN+ achieves 99.2% AUROC, significantly outperforming DPNrev (85.4%) and DPN− (99.7% AUROC but with only 3.5% precision).
• On CIFAR-100, DPN+ achieves 99.0% AUPR and 99.5% AUROC, surpassing DPNrev (85.0% AUROC) and DPN− (99.0% AUROC with 7.5% precision).
• On STL-10, DPN+ achieves 100.0% AUROC and 100.0% AUPR, outperforming DPNrev (99.5% AUROC) and DPN− (100.0% AUROC with 0.2% precision).
• On Places365, DPN+ achieves 100.0% AUROC and 100.0% AUPR, significantly exceeding DPNrev (99.1% AUROC) and DPN− (99.9% AUROC with 0.8% precision).
• On Textures, DPN+ achieves 98.4% AUROC and 97.9% AUPR, outperforming DPNrev (92.6% AUROC) and DPN− (98.7% AUROC with 19.3% precision).
• The method consistently achieves near-perfect OOD detection performance across all datasets, with DPN+ showing superior generalization and robustness compared to DPN−, which suffers from low precision and poor OOD distinction.