[Paper Review] MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks

Zhiqiang Shen, Marios Savvides|arXiv (Cornell University)|Sep 17, 2020

Image Processing Techniques and ApplicationsEngineering58 references51 citations

TL;DR

MEAL V2 uses an ensemble teacher distillation framework with final-output supervision and a good student initialization to push vanilla ResNet-50 beyond 80% Top-1 on ImageNet without architecture changes or extra data. It also improves compact models substantially.

ABSTRACT

We introduce a simple yet effective distillation framework that is able to boost the vanilla ResNet-50 to 80%+ Top-1 accuracy on ImageNet without tricks. We construct such a framework through analyzing the problems in the existing classification system and simplify the base method ensemble knowledge distillation via discriminators by: (1) adopting the similarity loss and discriminator only on the final outputs and (2) using the average of softmax probabilities from all teacher ensembles as the stronger supervision. Intriguingly, three novel perspectives are presented for distillation: (1) weight decay can be weakened or even completely removed since the soft label also has a regularization effect; (2) using a good initialization for students is critical; and (3) one-hot/hard label is not necessary in the distillation process if the weights are well initialized. We show that such a straight-forward framework can achieve state-of-the-art results without involving any commonly-used techniques, such as architecture modification; outside training data beyond ImageNet; autoaug/randaug; cosine learning rate; mixup/cutmix training; label smoothing; etc. Our method obtains 80.67% top-1 accuracy on ImageNet using a single crop-size of 224x224 with vanilla ResNet-50, outperforming the previous state-of-the-arts by a significant margin under the same network structure. Our result can be regarded as a strong baseline using knowledge distillation, and to our best knowledge, this is also the first method that is able to boost vanilla ResNet-50 to surpass 80% on ImageNet without architecture modification or additional training data. On smaller ResNet-18, our distillation framework consistently improves from 69.76% to 73.19%, which shows tremendous practical values in real-world applications. Our code and models are available at: https://github.com/szq0214/MEAL-V2.

Motivation & Objective

• Diagnose limitations of standard classification training and one-hot supervision.
• Propose a simple, effective knowledge distillation framework that boosts existing architectures without tricks.
• Show that soft labels, strong teacher ensembles, and good student initialization improve generalization.
• Demonstrate the method’s applicability to both large (ResNet-50) and compact models (MobileNet, EfficientNet) without data beyond ImageNet.

Proposed method

• Use a teacher ensemble to generate soft supervision by averaging softmax outputs of multiple pre-trained teachers.
• Train the student by minimizing KL-divergence between the student output and the ensembled teacher distribution (equivalent to cross-entropy with soft targets).
• Employ a discriminator to distinguish teacher vs. student features, providing a regularization signal.
• Adopt a single-stage distillation pipeline using final-layer supervision (no intermediate feature matching).
• Weaken or remove weight decay because soft labels provide regularization and a good initialization is crucial.
• Do not rely on architecture changes or external data; use standard ImageNet training with basic augmentation.

Experimental results

Research questions

• RQ1 Can knowledge distillation with a powerful teacher ensemble and soft supervision boost vanilla ResNet-50 to surpass 80% Top-1 accuracy on ImageNet without architectural changes or extra data?
• RQ2 How do initialization quality, learning rate strategy, and regularization choices affect distillation performance in MEAL V2?
• RQ3 Is the approach transferable to compact networks (MobileNet, EfficientNet) and to transfer learning tasks?
• RQ4 What is the role of the discriminator and target-layer supervision in stabilizing or improving distillation?
• RQ5 How does MEAL V2 compare to prior distillation methods in terms of mimicking ability and final accuracy?

Key findings

• ResNet-50 with MEAL V2 achieves 80.67% Top-1 on ImageNet with 224×224 input, outperforming the prior state-of-the-art MEAL by 2.46%.
• With 380×380 input, ResNet-50 reaches 81.72% Top-1, beating a baseline by 2.62%.
• MEAL V2 substantially improves compact models (e.g., MobileNet V3-Small 0.75/1.0 and Large 1.0) by about 2.2–2.25 percentage points without architecture changes.
• The method remains effective with modest data augmentation and even with additional CutMix, indicating not being saturated on ImageNet.
• Stronger teacher ensembles yield better student performance, and good student initialization critically improves outcomes.
• Ablations show removing weight decay and relying on soft labels can be advantageous when initialization is strong.