[Paper Review] Object-centric Sampling for Fine-grained Image Classification

Xiaoyu Wang, Tianbao Yang+2 more|arXiv (Cornell University)|Dec 10, 2014

Advanced Neural Network Applications12 references8 citations

TL;DR

This paper proposes object-centric sampling (OCS) for fine-grained image classification, using a saliency-aware object detector to guide CNN training by prioritizing image windows around detected objects. By constructing a large-scale car dataset with 333 classes and 157K training images, the method improves top-1 accuracy from 81.6% to 89.3% through multinomial sampling that emphasizes detection responses, effectively reducing background clutter and improving generalization on fine-grained distinctions.

ABSTRACT

This paper proposes to go beyond the state-of-the-art deep convolutional neural network (CNN) by incorporating the information from object detection, focusing on dealing with fine-grained image classification. Unfortunately, CNN suffers from over-fiting when it is trained on existing fine-grained image classification benchmarks, which typically only consist of less than a few tens of thousands training images. Therefore, we first construct a large-scale fine-grained car recognition dataset that consists of 333 car classes with more than 150 thousand training images. With this large-scale dataset, we are able to build a strong baseline for CNN with top-1 classification accuracy of 81.6%. One major challenge in fine-grained image classification is that many classes are very similar to each other while having large within-class variation. One contributing factor to the within-class variation is cluttered image background. However, the existing CNN training takes uniform window sampling over the image, acting as blind on the location of the object of interest. In contrast, this paper proposes an \emph{object-centric sampling} (OCS) scheme that samples image windows based on the object location information. The challenge in using the location information lies in how to design powerful object detector and how to handle the imperfectness of detection results. To that end, we design a saliency-aware object detection approach specific for the setting of fine-grained image classification, and the uncertainty of detection results are naturally handled in our OCS scheme. Our framework is demonstrated to be very effective, improving top-1 accuracy to 89.3% (from 81.6%) on the large-scale fine-grained car classification dataset.

Motivation & Objective

• To address the challenge of overfitting in fine-grained image classification due to limited training data in existing benchmarks.
• To reduce within-class variation caused by cluttered backgrounds in fine-grained images.
• To improve CNN performance by shifting from uniform sampling to object-centric sampling based on object detection.
• To design a robust detection and sampling pipeline that handles imperfect detection results.
• To establish a strong CNN baseline using a large-scale, high-resolution fine-grained dataset for car recognition.

Proposed method

• Construct a large-scale fine-grained car dataset with 333 classes and 157,023 training images, each labeled by brand, model, and year range.
• Design a saliency-aware object detection framework based on Regionlet detection, trained on a dataset where only the most salient object is labeled as positive.
• Use selective search to generate object proposals and apply Regionlet re-localization to refine bounding box predictions.
• Implement a multinomial sampling scheme that assigns higher sampling probability to image windows overlapping with high-confidence detection responses.
• Replace non-max suppression with a global max-operation to preserve detection confidence across the image.
• Train a deep CNN with 333 classes using both uniform sampling and multinomial sampling for ablation and comparison.

Experimental results

Research questions

• RQ1 Can a large-scale, fine-grained dataset significantly improve CNN performance in fine-grained image classification?
• RQ2 Does object-centric sampling based on detection results reduce the negative impact of background clutter in fine-grained classification?
• RQ3 How effective is a saliency-aware detector in identifying the most relevant object in complex, cluttered images?
• RQ4 Can a multinomial sampling strategy robustly handle imperfect detection outputs while improving classification accuracy?
• RQ5 To what extent does object-centric sampling outperform uniform sampling in fine-grained recognition tasks?

Key findings

• The proposed large-scale car dataset enables a strong CNN baseline with 81.6% top-1 accuracy, significantly outperforming conventional methods.
• Object-centric sampling using multinomial sampling based on detection responses improves top-1 accuracy to 89.3%, a relative gain of 7.7 percentage points.
• The saliency-aware detector achieves 85.8% average precision and consistently assigns higher confidence to larger, more salient objects.
• The multinomial sampling scheme effectively reduces reliance on background features, improving generalization on fine-grained distinctions.
• The framework demonstrates robustness to imperfect detection, as shown by consistent performance gains even with noisy detection outputs.
• Visualization confirms that the model focuses on the correct object region, with fewer false predictions when using OCS.