[Paper Review] Solving Long-tailed Recognition with Deep Realistic Taxonomic Classifier

Tz-Ying Wu, Pedro Morgado+3 more|arXiv (Cornell University)|Jul 20, 2020

Domain Adaptation and Few-Shot Learning42 references4 citations

TL;DR

This paper proposes Deep-RTC, a deep realistic taxonomic classifier that improves long-tailed recognition by allowing models to reject fine-grained classification for coarser, more confident predictions when uncertainty is high. By using stochastic tree sampling during training and dynamic inference with multi-level exit decisions, Deep-RTC achieves state-of-the-art performance on long-tailed datasets, outperforming prior methods by up to 11% in correctly predicted bits (CPB).

ABSTRACT

Long-tail recognition tackles the natural non-uniformly distributed data in real-world scenarios. While modern classifiers perform well on populated classes, its performance degrades significantly on tail classes. Humans, however, are less affected by this since, when confronted with uncertain examples, they simply opt to provide coarser predictions. Motivated by this, a deep realistic taxonomic classifier (Deep-RTC) is proposed as a new solution to the long-tail problem, combining realism with hierarchical predictions. The model has the option to reject classifying samples at different levels of the taxonomy, once it cannot guarantee the desired performance. Deep-RTC is implemented with a stochastic tree sampling during training to simulate all possible classification conditions at finer or coarser levels and a rejection mechanism at inference time. Experiments on the long-tailed version of four datasets, CIFAR100, AWA2, Imagenet, and iNaturalist, demonstrate that the proposed approach preserves more information on all classes with different popularity levels. Deep-RTC also outperforms the state-of-the-art methods in longtailed recognition, hierarchical classification, and learning with rejection literature using the proposed correctly predicted bits (CPB) metric.

Motivation & Objective

• Address the performance drop of deep learning models on rare (tail) classes in long-tailed data distributions.
• Overcome the limitations of flat classifiers that force all predictions at the finest granularity, even when confidence is low.
• Mimic human cognition by allowing classifiers to make coarser, more reliable predictions when fine-grained classification is uncertain.
• Introduce a new evaluation metric, correctly predicted bits (CPB), to better capture information recovery in hierarchical classification.
• Develop a training and inference framework that enables dynamic prediction across multiple levels of a taxonomic tree while maintaining high confidence.

Proposed method

• Propose a stochastic tree sampling (STS) regularization during training to simulate all possible cuts of the taxonomic tree, akin to dropout but for hierarchical structures.
• Implement a dynamic softmax classifier that can generate predictions at any node in the taxonomy, not just the leaves.
• Use a rejection mechanism at inference that allows examples to exit the tree at intermediate nodes when confidence falls below a threshold.
• Train the model to produce calibrated probability estimates at every level of the taxonomy to enable reliable decision-making on when to exit.
• Integrate label encoding and hierarchical loss functions that support multi-level prediction and knowledge transfer across classes.
• Optimize for the CPB metric during training, which prioritizes correct information recovery over strict leaf-level accuracy.

Experimental results

Research questions

• RQ1 Can a hierarchical classifier that allows coarser predictions improve performance on long-tailed recognition compared to standard flat classifiers?
• RQ2 How does dynamic exit at intermediate taxonomic levels affect information recovery and prediction confidence across different class popularity levels?
• RQ3 To what extent can stochastic tree sampling during training improve generalization and robustness in long-tailed settings?
• RQ4 Does a model that prioritizes correct partial predictions (via CPB) outperform state-of-the-art flat and hierarchical methods in long-tailed recognition?
• RQ5 Can a dynamic, confidence-aware classifier reduce performance degradation on few-shot classes without sacrificing accuracy on many-shot classes?

Key findings

• Deep-RTC achieves a 9% improvement in CPB over the state-of-the-art flat classifier on iNaturalist-LT and an 11% improvement on ImageNet-LT.
• On all four datasets (CIFAR100-LT, AWA2-LT, ImageNet-LT, iNaturalist-LT), Deep-RTC maintains high performance across all class popularity levels, unlike prior methods that degrade on many-shot classes.
• For few-shot classes, Deep-RTC achieves a rejection rate of approximately 50%, with over 90% hierarchical accuracy on rejected examples, indicating high confidence in coarser predictions.
• Deep-RTC outperforms a state-of-the-art flat realistic predictor (RP) in all rejection rate scenarios, especially on few-shot and medium-shot classes, due to its ability to soft-reject at intermediate nodes.
• The method achieves superior results in hierarchical classification benchmarks, even with a simpler architecture than competing methods that add complex branching structures.
• The CPB metric reveals that Deep-RTC recovers significantly more information per sample than prior approaches, particularly for low-popularity classes, by favoring correct intermediate-level predictions over incorrect leaf-level ones.