[Paper Review] Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization

Yibing Liu, Chris Xing Tian+3 more|arXiv (Cornell University)|Jun 5, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces Neuron Activation Coverage (NAC), a novel measure that quantifies how thoroughly neurons in a deep neural network are activated by in-distribution (InD) training data. By modeling neuron influence via KL divergence gradients, NAC enables state-of-the-art out-of-distribution (OOD) detection (10.60% FPR95 improvement on CIFAR-100) and a strong correlation between NAC and model generalization, outperforming standard validation metrics in selecting robust models.

ABSTRACT

The out-of-distribution (OOD) problem generally arises when neural networks encounter data that significantly deviates from the training data distribution, i.e., in-distribution (InD). In this paper, we study the OOD problem from a neuron activation view. We first formulate neuron activation states by considering both the neuron output and its influence on model decisions. Then, to characterize the relationship between neurons and OOD issues, we introduce the extit{neuron activation coverage} (NAC) -- a simple measure for neuron behaviors under InD data. Leveraging our NAC, we show that 1) InD and OOD inputs can be largely separated based on the neuron behavior, which significantly eases the OOD detection problem and beats the 21 previous methods over three benchmarks (CIFAR-10, CIFAR-100, and ImageNet-1K). 2) a positive correlation between NAC and model generalization ability consistently holds across architectures and datasets, which enables a NAC-based criterion for evaluating model robustness. Compared to prevalent InD validation criteria, we show that NAC not only can select more robust models, but also has a stronger correlation with OOD test performance.

Motivation & Objective

• To address the fundamental challenge of out-of-distribution (OOD) detection and generalization in deep learning, where models fail under distribution shift.
• To identify a more principled, neuron-level characterization of model behavior that captures the root causes of OOD sensitivity.
• To develop a simple yet effective metric—Neuron Activation Coverage (NAC)—that quantifies neuron activation patterns under InD data.
• To demonstrate that NAC improves both OOD detection and model generalization evaluation beyond existing InD validation criteria.
• To establish a strong, consistent correlation between NAC and model robustness across architectures and datasets.

Proposed method

• Formulate neuron activation states by combining neuron output with its influence on model decisions, using gradients of KL divergence between network output and a uniform vector.
• Introduce Neuron Activation Coverage (NAC) as a statistical measure of how frequently neuron states are activated by InD training data.
• Use NAC for uncertainty estimation (NAC-UE): lower coverage scores indicate higher uncertainty, enabling OOD detection.
• Use NAC for model evaluation (NAC-ME): integrate coverage distribution across all neurons to assess model robustness.
• Leverage coverage analysis principles from software testing to identify under-activated neurons as potential failure points under OOD inputs.
• Apply NAC to three benchmarks (CIFAR-10, CIFAR-100, ImageNet-1K) and the DomainBed benchmark for OOD generalization evaluation.

Experimental results

Research questions

• RQ1 Can neuron activation patterns under InD data effectively distinguish OOD inputs from InD inputs?
• RQ2 Does a neuron's activation coverage correlate with model generalization ability across different architectures and datasets?
• RQ3 Can NAC-based uncertainty estimation outperform existing OOD detection methods?
• RQ4 Can NAC-based model evaluation identify more robust models than standard InD validation metrics?
• RQ5 Is there a fundamental link between neuron coverage and model robustness under distribution shift?

Key findings

• NAC-UE achieves a 10.60% improvement in FPR95 and a 4.58% gain in AUROC over the competitive ViM method on CIFAR-100, establishing SOTA performance in OOD detection.
• On the PACS dataset, NAC-ME selects models with 69.16% average accuracy on OOD generalization, outperforming the validation-based method (65.04%) and showing stronger correlation with OOD test performance.
• On OfficeHome, NAC-ME achieves 47.86% average OOD accuracy with Vit-b16, compared to 40.42% for the validation baseline, demonstrating superior robustness selection.
• The positive correlation between NAC and model generalization holds consistently across architectures (ResNet, Vision Transformer) and datasets (PACS, OfficeHome, TerraInc).
• NAC-ME correlates more strongly with OOD test performance than standard InD validation criteria, enabling better model selection for OOD robustness.
• The method is effective across diverse models and datasets, with NAC-ME achieving 99.14% OOD accuracy on RN50 in the OfficeHome benchmark, outperforming the validation baseline (98.71%).