[Paper Review] Learning from Web Data: the Benefit of Unsupervised Object Localization

Xiaoxiao Sun, Liang Zheng+2 more|arXiv (Cornell University)|Dec 21, 2018

Advanced Image and Video Retrieval Techniques46 references4 citations

TL;DR

This paper proposes an unsupervised object localization method to reduce the distribution gap between web data and standard datasets, improving fine-grained image classification. By enforcing scale, localization, and label consistency constraints on web image proposals, the method de-biases web data, achieving state-of-the-art performance with up to 6.87% accuracy gain on Stanford Dogs and 5.3% on MIT Indoor67.

ABSTRACT

Annotating a large number of training images is very time-consuming. In this background, this paper focuses on learning from easy-to-acquire web data and utilizes the learned model for fine-grained image classification in labeled datasets. Currently, the performance gain from training with web data is incremental, like a common saying "better than nothing, but not by much". Conventionally, the community looks to correcting the noisy web labels to select informative samples. In this work, we first systematically study the built-in gap between the web and standard datasets, i.e. different data distributions between the two kinds of data. Then, in addition to using web labels, we present an unsupervised object localization method, which provides critical insights into the object density and scale in web images. Specifically, we design two constraints on web data to substantially reduce the difference of data distributions for the web and standard datasets. First, we present a method to control the scale, localization and number of objects in the detected region. Second, we propose to select the regions containing objects that are consistent with the web tag. Based on the two constraints, we are able to process web images to reduce the gap, and the processed web data is used to better assist the standard dataset to train CNNs. Experiments on several fine-grained image classification datasets confirm that our method performs favorably against the state-of-the-art methods.

Motivation & Objective

• To investigate why web data provides limited performance gains in deep learning despite its large scale and low cost.
• To identify and quantify the 'built-in gap'—differences in object scale, localization, density, and label quality—between web and standard datasets.
• To develop a method that reduces this distribution gap using unsupervised object localization with data-driven constraints.
• To demonstrate that de-biased web data significantly improves fine-tuned CNN performance on fine-grained classification benchmarks.

Proposed method

• Proposes a two-stage constraint mechanism: (1) form constraint to control object scale, localization, and number of objects in detected regions; (2) label consistency constraint to filter regions matching web tags.
• Applies unsupervised object localization to web images to generate candidate regions, then enforces geometric and semantic constraints to reduce distribution shift.
• Introduces $\Delta_{form}$ to suppress redundant or noisy regions based on spatial and scale regularity, improving localization quality.
• Introduces $\Delta_{label}$ to filter regions inconsistent with web tags, reducing noisy supervision.
• Uses the constrained web data to pre-train or fine-tune CNNs (e.g., ResNet-110) on standard datasets, improving generalization.
• Employs ablation studies and cross-dataset evaluation to validate the effectiveness of each component.

Experimental results

Research questions

• RQ1 To what extent does the distribution gap between web and standard datasets limit the performance gain from using web data?
• RQ2 How do differences in object scale, localization, density, and label quality contribute to this gap?
• RQ3 Can unsupervised object localization with geometric and semantic constraints effectively reduce the built-in gap?
• RQ4 Does the proposed method outperform state-of-the-art approaches in fine-grained image classification using web data?
• RQ5 How does the method perform across diverse datasets with varying complexity, such as object-centric (Food-101, Stanford Dogs) and scene-based (MIT Indoor67) tasks?

Key findings

• The built-in gap between web and standard datasets—due to irregular object scale, localization, density, and label noise—significantly limits performance gains from web data.
• On Food-101, the proposed method improves classification accuracy by 6.25% over direct use of web data, achieving 84.31% accuracy with ResNet-110.
• On Stanford Dogs, the method achieves a 6.87% accuracy gain (from 81.26% to 88.13%) after de-biasing web data, outperforming baseline web data usage.
• On MIT Indoor67, the method improves accuracy by 5.3% (from 79.63% to 84.93%), demonstrating effectiveness even in complex scene recognition tasks.
• The ablation study confirms that both $\Delta_{form}$ and $\Delta_{label}$ constraints are essential, with each contributing significantly to performance improvement.
• The method achieves state-of-the-art results using only 1/3 of the data compared to prior work (e.g., Goldfince [4]), showing high data efficiency.