[Paper Review] WebVision Database: Visual Learning and Understanding from Web Data

Wen Li, Limin Wang+3 more|arXiv (Cornell University)|Aug 9, 2017

Domain Adaptation and Few-Shot Learning32 references313 citations

TL;DR

Introduces WebVision, a 2.4M-image web dataset with meta information to study learning from noisy web data for visual recognition and domain adaptation, showing competitive generalization to ILSVRC 2012 and strong transfer to Caltech-256 and PASCAL VOC 2007.

ABSTRACT

In this paper, we present a study on learning visual recognition models from large scale noisy web data. We build a new database called WebVision, which contains more than $2.4$ million web images crawled from the Internet by using queries generated from the 1,000 semantic concepts of the benchmark ILSVRC 2012 dataset. Meta information along with those web images (e.g., title, description, tags, etc.) are also crawled. A validation set and test set containing human annotated images are also provided to facilitate algorithmic development. Based on our new database, we obtain a few interesting observations: 1) the noisy web images are sufficient for training a good deep CNN model for visual recognition; 2) the model learnt from our WebVision database exhibits comparable or even better generalization ability than the one trained from the ILSVRC 2012 dataset when being transferred to new datasets and tasks; 3) a domain adaptation issue (a.k.a., dataset bias) is observed, which means the dataset can be used as the largest benchmark dataset for visual domain adaptation. Our new WebVision database and relevant studies in this work would benefit the advance of learning state-of-the-art visual models with minimum supervision based on web data.

Motivation & Objective

• Assess how noisy web labels impact visual recognition compared to human-annotated data.
• Evaluate generalization of models trained on WebVision to other datasets and tasks.
• Explore the usefulness of accompanying meta information in web images for recognition tasks.
• Investigate dataset bias between WebVision and ILSVRC 2012 and its implications for domain adaptation.

Proposed method

• Construct a WebVision dataset with 2.4M images from Flickr and Google Image Search using 1,000 ILSVRC 2012 synsets as queries.
• Collect meta information (title, description, tags, etc.) for web images.
• Create a 100K human-annotated subset (50K validation, 50K test) via AMT, with near-duplicate removal and 3-vote quality voting.
• Train baseline AlexNet models on WebVision and ILSVRC 2012, and compare cross-dataset performance on validation sets.
• Evaluate transfer learning by applying WebVision- and ILSVRC-trained features to Caltech-256, PASCAL VOC 2007, and object detection with Faster R-CNN.
• Analyze label noise and quantity vs. quality by subsampling WebVision and ILSVRC 2012 images to study effects on recognition performance.

Experimental results

Research questions

• RQ1 Can noisy web-labeled data train competitive visual recognition models compared to human-annotated datasets?
• RQ2 How does model performance trained on WebVision generalize to other datasets and tasks (transfer learning)?
• RQ3 What is the impact of label noise versus data quantity in web-sourced training data?
• RQ4 Does meta information accompanying web images improve recognition performance or enable multi-modal learning?
• RQ5 Is there a measurable dataset bias between WebVision and ILSVRC 2012, and can WebVision serve as a benchmark for domain adaptation?

Key findings

• WebVision enables robust CNN models despite significant label noise, with large-scale data mitigating noise impact.
• Models trained on WebVision generalize comparably or better than ILSVRC 2012 models on Caltech-256 and PASCAL VOC 2007, and object detection on PASCAL VOC 2007.
• Domain bias exists between WebVision and ILSVRC 2012, as shown by cross-dataset performance drops, yet WebVision features transfer well to other tasks.
• Meta information associated with web images has potential to support multi-modal and domain adaptation studies (demonstrated by dataset bias observations).
• Using more web images can compensate label noise better than improving label quality alone; quantity benefits outweigh noise in large-scale settings.