[Paper Review] Image Classification and Retrieval from User-Supplied Tags

Hamid Izadinia, Ali Farhadi+2 more|arXiv (Cornell University)|Nov 25, 2014

Advanced Image and Video Retrieval Techniques21 references4 citations

TL;DR

This paper proposes a robust classification framework that directly learns from noisy, unfiltered user-supplied tags on Flickr, using a stochastic EM-based robust logistic regression to handle label noise and a calibration procedure with minimal manual annotations. It demonstrates that training on freely available user tags alone achieves performance comparable to costly manual annotation, and with calibration, surpasses it at 1/200th the labeling cost.

ABSTRACT

This paper proposes direct learning of image classification from user-supplied tags, without filtering. Each tag is supplied by the user who shared the image online. Enormous numbers of these tags are freely available online, and they give insight about the image categories important to users and to image classification. Our approach is complementary to the conventional approach of manual annotation, which is extremely costly. We analyze of the Flickr 100 Million Image dataset, making several useful observations about the statistics of these tags. We introduce a large-scale robust classification algorithm, in order to handle the inherent noise in these tags, and a calibration procedure to better predict objective annotations. We show that freely available, user-supplied tags can obtain similar or superior results to large databases of costly manual annotations.

Motivation & Objective

• To address the high cost and limited scalability of manually curated image annotation datasets.
• To investigate whether unfiltered, user-supplied tags from online photo-sharing platforms can serve as a viable alternative to expensive manual labeling.
• To develop a robust learning method that handles the inherent noise, inconsistency, and variability in user-provided tags.
• To calibrate model predictions using a small set of manual annotations to improve performance on objective image annotations.
• To evaluate the method on image classification, tag prediction, and multi-tag image retrieval tasks.

Proposed method

• Proposes a robust logistic regression (RLR) model that jointly learns classifier weights and tag-specific outlier probabilities to handle randomly omitted positive labels.
• Employs a stochastic Expectation-Maximization (EM) algorithm to scale the RLR to large datasets like the Flickr 100 Million Image dataset.
• Introduces a calibration procedure that adjusts model output probabilities using a small, manually annotated subset (e.g., NUS-WIDE) to align predictions with objective annotations.
• Uses the F100M dataset for pre-training on user-supplied tags and NUS-WIDE for calibration and evaluation on objective labels.
• Applies the calibrated model to predict tags, objective annotations, and retrieve images based on multi-tag queries.
• Evaluates performance using precision, recall, and F1-score averaged over tags, with normalized precision at 5 for retrieval tasks.

Experimental results

Research questions

• RQ1 Can user-supplied tags from photo-sharing platforms be effectively used for large-scale image classification without manual curation?
• RQ2 How does robust logistic regression with stochastic EM improve classification performance under noisy, unfiltered tags?
• RQ3 Can a small set of manual annotations significantly improve model performance when combined with large-scale user-tag pre-training?
• RQ4 Does the proposed method outperform traditional manual annotation approaches in terms of accuracy and labeling cost efficiency?
• RQ5 How effective is the model in multi-tag image retrieval tasks compared to standard logistic regression?

Key findings

• Training on the Flickr 100 Million (F100M) dataset using user-supplied tags alone achieves comparable performance to models trained on smaller, manually annotated datasets like NUS-WIDE.
• Calibrating the F100M-trained model with just 1/200th of the manual annotations yields superior performance on objective annotation prediction compared to training on the full manual dataset.
• Robust logistic regression (RLR) consistently outperforms standard logistic regression (LR) in both tag prediction and image retrieval, especially for longer multi-tag queries.
• The calibrated RLR model achieves a mean F1-score of 71.9 on the NUS-WIDE dataset for tag prediction, outperforming state-of-the-art methods on that benchmark.
• For multi-tag image retrieval, the calibrated RLR model achieves a normalized precision at 5 of 11.0 for two-tag queries, significantly outperforming standard LR and other baselines.
• The method enables high-quality image classification and retrieval for thousands of tags even when no manual annotations exist, by leveraging the scale and diversity of user-supplied tags.