[Paper Review] ImageNet-21K Pretraining for the Masses

Tal Ridnik, Emanuel Ben-Baruch+2 more|arXiv (Cornell University)|Apr 22, 2021

Multimodal Machine Learning Applications57 references4 citations

TL;DR

This paper presents a novel, efficient, and accessible pretraining pipeline for ImageNet-21K, leveraging WordNet's hierarchical structure to transform single-label annotations into semantic multi-labels. By introducing a new training scheme called semantic softmax, the method significantly outperforms both ImageNet-1K and prior ImageNet-21K pretraining approaches across diverse models—including small mobile architectures—achieving state-of-the-art results on ImageNet-1K and multiple downstream tasks using only publicly available data.

ABSTRACT

ImageNet-1K serves as the primary dataset for pretraining deep learning models for computer vision tasks. ImageNet-21K dataset, which is bigger and more diverse, is used less frequently for pretraining, mainly due to its complexity, low accessibility, and underestimation of its added value. This paper aims to close this gap, and make high-quality efficient pretraining on ImageNet-21K available for everyone. Via a dedicated preprocessing stage, utilization of WordNet hierarchical structure, and a novel training scheme called semantic softmax, we show that various models significantly benefit from ImageNet-21K pretraining on numerous datasets and tasks, including small mobile-oriented models. We also show that we outperform previous ImageNet-21K pretraining schemes for prominent new models like ViT and Mixer. Our proposed pretraining pipeline is efficient, accessible, and leads to SoTA reproducible results, from a publicly available dataset. The training code and pretrained models are available at: https://github.com/Alibaba-MIIL/ImageNet21K

Motivation & Objective

• To address the underutilization of ImageNet-21K despite its larger scale and diversity compared to ImageNet-1K.
• To overcome challenges in pretraining on ImageNet-21K, including non-exclusive labels, lack of standardized splits, and high computational cost.
• To develop a methodologically sound, efficient, and reproducible pretraining pipeline that makes high-quality ImageNet-21K pretraining accessible to all practitioners.
• To demonstrate that even small, mobile-optimized models benefit significantly from ImageNet-21K pretraining, challenging the assumption that only large models gain from large-scale pretraining.

Proposed method

• Preprocess ImageNet-21K by standardizing image resolution, creating a train-validation split, and ensuring class-level image balance.
• Use WordNet’s synset hierarchy to convert each single label into a set of semantic multi-labels, representing hierarchical relationships.
• Design a novel training scheme called semantic softmax, which applies separate softmax operations across multiple levels of the WordNet hierarchy and aggregates losses with hierarchy balancing.
• Integrate semantic softmax into a dedicated knowledge distillation loss to further improve feature representation and downstream performance.
• Apply the pipeline to a wide range of models, including ViT, ResNet, MobileNetV3, and OFA-595, across diverse downstream tasks.
• Ensure reproducibility by releasing training code and pretrained models via a public GitHub repository.

Experimental results

Research questions

• RQ1 Can a standardized, efficient, and accessible pretraining pipeline be built for ImageNet-21K to overcome its historical underuse despite its scale and quality?
• RQ2 Does multi-label training on ImageNet-21K, using semantic hierarchies, improve downstream performance compared to single-label training?
• RQ3 Can the proposed semantic softmax method effectively exploit the hierarchical structure of ImageNet-21K labels to outperform both single-label and multi-label pretraining?
• RQ4 Do small, mobile-optimized models benefit from ImageNet-21K pretraining, challenging the notion that only large models can leverage large-scale datasets?
• RQ5 Can ImageNet-21K pretraining achieve state-of-the-art performance on ImageNet-1K without relying on private datasets?

Key findings

• The proposed semantic softmax pretraining scheme significantly outperforms standard ImageNet-1K pretraining across all tested models, including MobileNetV3 and OFA-595, with a 2.9% absolute improvement on iNaturalist.
• ImageNet-21K pretraining with semantic softmax achieves a new state-of-the-art top-1 accuracy of 78.0% on ImageNet-1K using MobileNetV3, surpassing the original reported 75.2%.
• For ResNet50, the method achieves 82.0% top-1 accuracy on ImageNet-1K, a 6.0% improvement over the original 76.0% reported in prior work.
• The semantic softmax approach improves training stability and performance on MLP-based models like ViT and Mixer, which are typically harder to fine-tune due to lack of inductive bias.
• Even small models benefit from ImageNet-21K pretraining, demonstrating that large-scale pretraining is not limited to large architectures.
• The method achieves SoTA results on multiple downstream tasks—including action recognition and Pascal-VOC—without using private datasets, using only publicly available data and code.