[Paper Review] Adversarial-Based Knowledge Distillation for Multi-Model Ensemble and Noisy Data Refinement

Zhiqiang Shen, Zhankui He+5 more|arXiv (Cornell University)|Aug 22, 2019

Advanced Neural Network Applications4 citations

TL;DR

This paper proposes MEAL, a novel adversarial-based knowledge distillation framework that compresses multi-model ensembles into a single student network while refining noisy labels. By leveraging soft labels from diverse teacher models and adversarial training to align student features with teacher distributions, MEAL achieves state-of-the-art performance—outperforming the original ResNet-50 by 2.06% top-1 error on ImageNet and improving top-3 accuracy by 1.15% on iMaterialist—while enabling fast single-forward-pass inference.

ABSTRACT

Generic Image recognition is a fundamental and fairly important visual problem in computer vision. One of the major challenges of this task lies in the fact that single image usually has multiple objects inside while the labels are still one-hot, another one is noisy and sometimes missing labels when annotated by humans. In this paper, we focus on tackling these challenges accompanying with two different image recognition problems: multi-model ensemble and noisy data recognition with a unified framework. As is well-known, usually the best performing deep neural models are ensembles of multiple base-level networks, as it can mitigate the variation or noise containing in the dataset. Unfortunately, the space required to store these many networks, and the time required to execute them at runtime, prohibit their use in applications where test sets are large (e.g., ImageNet). In this paper, we present a method for compressing large, complex trained ensembles into a single network, where the knowledge from a variety of trained deep neural networks (DNNs) is distilled and transferred to a single DNN. In order to distill diverse knowledge from different trained (teacher) models, we propose to use adversarial-based learning strategy where we define a block-wise training loss to guide and optimize the predefined student network to recover the knowledge in teacher models, and to promote the discriminator network to distinguish teacher vs. student features simultaneously. Extensive experiments on CIFAR-10/100, SVHN, ImageNet and iMaterialist Challenge Dataset demonstrate the effectiveness of our MEAL method. On ImageNet, our ResNet-50 based MEAL achieves top-1/5 21.79%/5.99% val error, which outperforms the original model by 2.06%/1.14%. On iMaterialist Challenge Dataset, our MEAL obtains a remarkable improvement of top-3 1.15% (official evaluation metric) on a strong baseline model of ResNet-101.

Motivation & Objective

• To address the high computational and memory cost of multi-model ensembles in real-world applications.
• To improve robustness and accuracy in image recognition under noisy or ambiguous annotations.
• To compress knowledge from diverse pre-trained teacher networks into a single efficient student model.
• To leverage soft label distributions from multiple architectures to provide richer supervision than one-hot labels.
• To develop a unified framework that enhances model generalization and handles multi-object and noisy label challenges.

Proposed method

• The method uses a student network trained to mimic the soft label outputs of multiple pre-trained teacher models, replacing one-hot labels with diverse, informative distributions.
• A block-wise adversarial training strategy is introduced, where a discriminator distinguishes between features from teacher and student networks to enforce feature-level alignment.
• The student network is optimized via a combined loss: knowledge distillation loss from soft labels and adversarial loss from the discriminator to improve feature-level generalization.
• Iterative refinement is applied by retraining the student with updated teacher models, progressively improving supervision quality.
• The framework supports end-to-end training with diverse architectures (e.g., ResNet, DenseNet, VGG) to generate diverse and informative supervision signals.
• The method enables fast inference via a single forward pass, eliminating the need for ensemble inference at test time.

Experimental results

Research questions

• RQ1 Can adversarial knowledge distillation effectively compress multi-model ensembles into a single efficient student network without sacrificing performance?
• RQ2 Can soft label distributions from diverse teacher models improve robustness to noisy and multi-object annotations?
• RQ3 Does adversarial feature matching between teacher and student networks lead to better generalization than standard distillation?
• RQ4 How does iterative refinement of teacher-student pairs affect model accuracy and robustness on noisy datasets?
• RQ5 To what extent does the diversity of supervision from different architectures enhance student model performance?

Key findings

• On ImageNet, the ResNet-50-based MEAL achieved a top-1 validation error of 21.79% and top-5 error of 5.99%, outperforming the original model by 2.06 and 1.14 percentage points respectively.
• On the iMaterialist Products dataset, MEAL improved the official top-3 error by 1.15% over a strong ResNet-101 baseline, demonstrating effectiveness on complex, real-world data.
• With data augmentation (CutMix and cosine learning rate), MEAL further reduced test error to 6.89% on the noisy CIFAR-10 dataset, indicating robustness to overfitting.
• Iterative retraining improved performance progressively, with the first round yielding the largest gain, confirming the effectiveness of refining supervision signals.
• The method achieved optimal performance with 4 teacher models on CIFAR-10, and training converged fully by ~500 epochs under increased budget.
• Visualization of softmax outputs showed that weaker models (e.g., SqueezeNet, VGG) produced more diverse label distributions than stronger ones (e.g., ResNet, DenseNet), validating the diversity of supervision.