[Paper Review] The Devil is in the Tails: Fine-grained Classification in the Wild

Grant Van Horn, Pietro Perona|arXiv (Cornell University)|Sep 5, 2017

Domain Adaptation and Few-Shot Learning34 references137 citations

TL;DR

This paper analyzes how long-tailed real-world data (eBird, iNaturalist, and Pasadena Trees) affects fine-grained classification with a state-of-the-art network, showing strong head-class performance but severe tail-class deterioration and minimal transfer from head to tail.

ABSTRACT

The world is long-tailed. What does this mean for computer vision and visual recognition? The main two implications are (1) the number of categories we need to consider in applications can be very large, and (2) the number of training examples for most categories can be very small. Current visual recognition algorithms have achieved excellent classification accuracy. However, they require many training examples to reach peak performance, which suggests that long-tailed distributions will not be dealt with well. We analyze this question in the context of eBird, a large fine-grained classification dataset, and a state-of-the-art deep network classification algorithm. We find that (a) peak classification performance on well-represented categories is excellent, (b) given enough data, classification performance suffers only minimally from an increase in the number of classes, (c) classification performance decays precipitously as the number of training examples decreases, (d) surprisingly, transfer learning is virtually absent in current methods. Our findings suggest that our community should come to grips with the question of long tails.

Motivation & Objective

• Highlight how long-tailed (head-tail) distributions affect fine-grained visual classification in the wild.
• Quantify performance gaps between well-represented head classes and sparsely represented tail classes.
• Assess transfer learning and cross-class knowledge transfer within a single-domain long-tail setting.

Proposed method

• Use Inception-v3 pretrained on ImageNet and fine-tune all layers on various long-tail and uniform datasets derived from eBird.
• Construct uniform, approximate long-tail, and full long-tail datasets with controlled head/tail splits and image counts per class.
• Compare training regimes including uniform vs natural sampling, and joint head-tail vs isolated head or tail model training.
• Evaluate using top-1 accuracy on validation/test splits with center crops at test time.
• Analyze transfer effects by varying head/tail data and reporting head-only vs tail-only vs joint training results.

Experimental results

Research questions

• RQ1 How does long-tail distribution of training images affect fine-grained classification within a single domain?
• RQ2 What is the impact of head versus tail data on overall and tail-class accuracy when training a single model?
• RQ3 Is there significant transfer learning or knowledge transfer from well-represented head classes to sparsely represented tail classes?
• RQ4 Do sampling strategies (uniform vs natural distribution) influence tail and head class performance in long-tail settings?

Key findings

• Fine-grained accuracy is excellent when each class has thousands of images, but degrades dramatically as tail class examples diminish.
• With sufficient data, increasing the number of classes harms performance only slightly; data quantity drives accuracy more than class count.
• Transfer learning between head and tail classes is negligible in current models; adding head-class data does not meaningfully improve tail-class performance within a single domain.
• Head classes benefit from natural sampling; tail classes show limited or no transfer benefits from head-class data when trained jointly.
• Tail performance remains poor even when head classes are trained with abundant data, indicating the need for low-shot and transfer-learning approaches.