[Paper Review] On Model Calibration for Long-Tailed Object Detection and Instance Segmentation

Tai-Yu Pan, Cheng Zhang+6 more|arXiv (Cornell University)|Jul 5, 2021

Advanced Neural Network ApplicationsComputer Science69 references27 citations

TL;DR

NorCal introduces a post-processing calibration method that reweights per-class scores by training sample size and normalizes scores across classes, improving performance on long-tailed LVIS without retraining.

ABSTRACT

Vanilla models for object detection and instance segmentation suffer from the heavy bias toward detecting frequent objects in the long-tailed setting. Existing methods address this issue mostly during training, e.g., by re-sampling or re-weighting. In this paper, we investigate a largely overlooked approach -- post-processing calibration of confidence scores. We propose NorCal, Normalized Calibration for long-tailed object detection and instance segmentation, a simple and straightforward recipe that reweighs the predicted scores of each class by its training sample size. We show that separately handling the background class and normalizing the scores over classes for each proposal are keys to achieving superior performance. On the LVIS dataset, NorCal can effectively improve nearly all the baseline models not only on rare classes but also on common and frequent classes. Finally, we conduct extensive analysis and ablation studies to offer insights into various modeling choices and mechanisms of our approach. Our code is publicly available at https://github.com/tydpan/NorCal/.

Motivation & Objective

• Motivate and address long-tailed biases in object detection and instance segmentation beyond training-time techniques.
• Propose a simple, model-agnostic post-processing calibration method (NorCal) that operates on pre-trained detectors.
• Demonstrate that score normalization across classes combined with class-size-based reweighting improves detection and segmentation across rare, common, and frequent classes on LVIS.
• Show that NorCal can improve both AP and AP Fixed metrics, indicating better ranking within each class without sacrificing frequent-class performance.

Proposed method

• Apply post-processing calibration to the classification sub-network of a pre-trained detector.
• Re-scale the logit of each foreground class by a factor a_c that depends on the class training size N_c, using a_c = N_c^gamma with a single hyper-parameter gamma.
• Decompose the classification probability to separately model foreground vs background, keeping the background logit intact.
• Normalize the adjusted class scores across all foreground classes (including background in the denominator) to re-rank proposals within each class.
• Optionally extend NorCal to multiple binary sigmoid classifiers by calibrating the exponentials or probabilities rather than logits, while noting normalization challenges.
• Tune gamma on training data, requiring only one hyper-parameter and no held-out validation set.

Experimental results

Research questions

• RQ1 Can post-processing calibration of class confidences, tuned by training-set class frequencies, reduce head-biased predictions in long-tailed detectors?
• RQ2 Does normalizing calibrated scores across all classes—and properly handling the background class—improve ranking and metrics for rare, common, and frequent classes on LVIS?
• RQ3 Is NorCal model-agnostic and effective across detectors with softmax and multiple binary classifiers?
• RQ4 How does NorCal compare to existing post-calibration methods in terms of robustness and performance across various LVIS-based tasks?

Key findings

• NorCal consistently improves baseline detectors on LVIS v1 for both object detection and instance segmentation across rare, common, and frequent classes.
• Score normalization across classes is crucial; without normalization, gains on rare classes are not realized and frequent-class performance may drop.
• A single hyper-parameter gamma controls calibration strength and can be robustly tuned using training data.
• Compared to other post-calibration methods, NorCal achieves larger gains with one hyper-parameter and explicit score normalization.
• NorCal improves both AP and AP Fixed metrics, indicating improved ranking within each class without sacrificing frequent-class performance.