[Paper Review] Label Refinery: Improving ImageNet Classification through Label Progression

Hessam Bagherinezhad, Maxwell Horton+2 more|arXiv (Cornell University)|May 7, 2018

Machine Learning and Data Classification24 references155 citations

TL;DR

The paper introduces Label Refinery, an iterative method that refines crop-level labels to soft, dynamic targets, yielding substantial accuracy gains across many architectures on ImageNet.

ABSTRACT

Among the three main components (data, labels, and models) of any supervised learning system, data and models have been the main subjects of active research. However, studying labels and their properties has received very little attention. Current principles and paradigms of labeling impose several challenges to machine learning algorithms. Labels are often incomplete, ambiguous, and redundant. In this paper we study the effects of various properties of labels and introduce the Label Refinery: an iterative procedure that updates the ground truth labels after examining the entire dataset. We show significant gain using refined labels across a wide range of models. Using a Label Refinery improves the state-of-the-art top-1 accuracy of (1) AlexNet from 59.3 to 67.2, (2) MobileNet from 70.6 to 73.39, (3) MobileNet-0.25 from 50.6 to 55.59, (4) VGG19 from 72.7 to 75.46, and (5) Darknet19 from 72.9 to 74.47.

Motivation & Objective

• Identify limitations of conventional single-label annotations in image classification.
• Propose an iterative, data-driven label refinement process that produces soft, dynamic, and collective labels.
• Demonstrate consistent accuracy gains across diverse architectures on ImageNet.
• Analyze factors influencing refinery quality (architecture, loss, dynamics, and adversarial labeling).

Proposed method

• Formalize label augmentation for crops within a dataset using sequential Label Refinery networks.
• Train the first refinery Cθ1 with standard cross-entropy against ground-truth labels.
• Train subsequent refiners Cθt (t>1) by minimizing KL-divergence between its output and the previous refiner's soft labels, effectively propagating information through iterations.
• Optionally use adversarial jittering by modifying inputs to maximize KL-divergence between successive refiners.
• Experiment with different refinery architectures to study cross-architecture benefits and overfitting behavior.
• Compare image-level vs category-level label refinement to assess where refinements yield the most benefit.

Experimental results

Research questions

• RQ1 Can iterative, crop-level label refinement using a model-generated soft labeling scheme improve ImageNet classification across multiple architectures?
• RQ2 How do dynamic (image-specific) vs soft (probabilistic) labels contribute to performance gains?
• RQ3 Does a refinery trained on one architecture effectively improve another architecture (cross-architecture refinement)?
• RQ4 What is the impact of using adversarially perturbed crops in the refinery process?

Key findings

• Refinery-based training consistently improves Top-1 accuracy across architectures (e.g., AlexNet, VGG19, ResNet50, MobileNet, Darknet19).
• Soft dynamic labeling and crop-wise label refinement provide complementary gains, with combined use yielding further improvements.
• Cross-architecture refinement (e.g., using ResNet-50 as a refinery for other nets) yields significant accuracy boosts, especially for compact models like MobileNet.
• Self-refinement and sequential refiners reduce overfitting, aligning train and validation performance and improving generalization.
• Adversarially generated labels (adversarial jittering) further boost accuracy for smaller models when used with refinement.
• Category-level taxonomy-based refinement does not outperform image-level refinement, suggesting visual similarity drives gains more than semantic taxonomies.
• Choice of refinery architecture matters; less overfitting refinery models (even with lower validation accuracy) can produce higher-quality crop labels and better downstream performance.
• Using KL-divergence as the refinement loss generalizes beyond simple cross-entropy and supports various loss configurations.