[Paper Review] From ImageNet to Image Classification: Contextualizing Progress on Benchmarks

Dimitris Tsipras, Shibani Santurkar+3 more|arXiv (Cornell University)|May 22, 2020

Mobile Crowdsensing and Crowdsourcing61 citations

TL;DR

The paper analyzes biases and misalignment in ImageNet’s data collection pipeline using large-scale human annotations, revealing multi-object images, label-validation biases, and the need for human-centered evaluation alongside standard accuracy.

ABSTRACT

Building rich machine learning datasets in a scalable manner often necessitates a crowd-sourced data collection pipeline. In this work, we use human studies to investigate the consequences of employing such a pipeline, focusing on the popular ImageNet dataset. We study how specific design choices in the ImageNet creation process impact the fidelity of the resulting dataset---including the introduction of biases that state-of-the-art models exploit. Our analysis pinpoints how a noisy data collection pipeline can lead to a systematic misalignment between the resulting benchmark and the real-world task it serves as a proxy for. Finally, our findings emphasize the need to augment our current model training and evaluation toolkit to take such misalignments into account. To facilitate further research, we release our refined ImageNet annotations at https://github.com/MadryLab/ImageNetMultiLabel.

Motivation & Objective

• Assess how ImageNet's crowd-sourced collection process may introduce biases and misalignment with real-world object recognition tasks.
• Quantify the prevalence and impact of issues such as multi-object images and label-validation biases on model evaluation.
• Propose a refined annotation pipeline to better capture ground truth and study its effects on model performance.
• Advocate for evaluation metrics that incorporate human judgments to complement fixed-label accuracy on ImageNet.

Proposed method

• Develop a two-phase annotation pipeline (Candidate Labels via model predictions; Classify task for per-object labeling) to obtain fine-grained image annotations.
• Use top-5 model predictions from multiple models to generate candidate labels for each image.
• Employ a Contains task to filter candidate labels and a subsequent Classify task to assign per-object labels and a main object.
• Aggregate annotator responses with majority voting to infer per-image object counts and main labels.
• Analyze 10,000 ImageNet validation images (10 per class) to assess benchmark-task alignment.
• Compare model predictions to human-annotated ground truth using human-centric metrics beyond fixed-image-label accuracy.

Experimental results

Research questions

• RQ1 How aligned are ImageNet labels with the ground-truth main objects and object counts in images?
• RQ2 How do common data collection and validation choices bias the resulting annotations and model evaluation?
• RQ3 What is the impact of multi-object images and label ambiguities on standard top-1/top-5 accuracy?
• RQ4 Can human-in-the-loop evaluation reveal progress not captured by traditional accuracy metrics?

Key findings

• More than one-fifth of ImageNet images contain objects from multiple classes, indicating multi-object images are common in the benchmark.
• Model accuracy on multi-object images drops significantly when evaluated with a single ImageNet label, but multi-label evaluation reduces the gap.
• Annotators often validate multiple mutually exclusive labels during the Contains task, revealing bias in the validation process.
• Humans sometimes disagree with the ImageNet main object, showing misalignment between dataset labels and perceived main objects.
• A more natural evaluation treats predictions as correct if they match any object in the image, or the main object per human annotation, mitigating some biases.
• State-of-the-art models predict labels very close to ImageNet labels from a human perspective, suggesting improvements may reflect distribution-fitting rather than true ground-truth advancement.