[Paper Review] Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation from a Blackbox Model

Dongdong Wang, Yandong Li+2 more|arXiv (Cornell University)|Mar 31, 2020

Domain Adaptation and Few-Shot Learning49 references4 citations

TL;DR

This paper proposes Active Mixup, a data- and query-efficient knowledge distillation method that uses mixup-augmented synthetic images and active learning to train a student neural network from a blackbox teacher model with minimal real data and teacher queries. The approach achieves performance on par with or better than whitebox distillation methods, demonstrating that blackbox teachers are more productive than human raters due to their ability to label semantically meaningless mixup images.

ABSTRACT

We study how to train a student deep neural network for visual recognition by distilling knowledge from a blackbox teacher model in a data-efficient manner. Progress on this problem can significantly reduce the dependence on large-scale datasets for learning high-performing visual recognition models. There are two major challenges. One is that the number of queries into the teacher model should be minimized to save computational and/or financial costs. The other is that the number of images used for the knowledge distillation should be small; otherwise, it violates our expectation of reducing the dependence on large-scale datasets. To tackle these challenges, we propose an approach that blends mixup and active learning. The former effectively augments the few unlabeled images by a big pool of synthetic images sampled from the convex hull of the original images, and the latter actively chooses from the pool hard examples for the student neural network and query their labels from the teacher model. We validate our approach with extensive experiments.

Motivation & Objective

• To address the challenge of distilling knowledge from a blackbox teacher model with minimal labeled data and low query cost.
• To reduce reliance on large-scale datasets for training high-performing visual recognition models.
• To improve data efficiency and query efficiency in knowledge distillation when the teacher model is inaccessible or expensive to query.
• To validate that synthetic mixup images can effectively substitute for real data in distillation, even when semantically meaningless.
• To demonstrate that blackbox teacher models outperform human raters in teaching student networks due to their ability to label synthetic, ambiguous images.

Proposed method

• Generate a large pool of synthetic images using mixup, which creates convex combinations of existing unlabeled images with varying coefficients.
• Use active learning to select the most uncertain mixup images—prioritizing diversity by discarding duplicates from the same original image pair—based on the student network's predictions.
• Query the blackbox teacher model only for the selected high-uncertainty mixup images, treating their outputs as pseudo-labels for training the student.
• Iteratively retrain the student network using the newly queried labels, refining uncertainty estimates and improving performance over multiple rounds.
• Enforce a diversity constraint (C₂ in Eq. 3) to avoid redundant queries by ensuring only the highest-uncertainty mixup image from each original image pair is selected.
• Use the resulting pseudo-labeled mixup data to train the student model, enabling it to generalize well to real test images.

Experimental results

Research questions

• RQ1 Can mixup-augmented synthetic images effectively serve as training data for distilling knowledge from a blackbox teacher model?
• RQ2 Does combining mixup with active learning significantly reduce the number of queries to a blackbox teacher while maintaining or improving student performance?
• RQ3 How does the performance of distillation from a blackbox teacher compare to that from a whitebox teacher when using minimal real data?
• RQ4 To what extent can out-of-domain data (e.g., CIFAR-100 images) be used to generate effective synthetic training data for distillation on a target domain (e.g., CIFAR-10)?
• RQ5 Can a student model trained on semantically meaningless mixup images still achieve high accuracy on real test images?

Key findings

• Active Mixup achieves 83.03% top-1 accuracy on CIFAR-10 using only 2,000 out-of-domain images and 80K synthetic mixup images, matching 87.89% accuracy from 80K real images in the in-domain setting.
• With 40K selected synthetic images and 2,000 real images, the method achieves 77.89% accuracy, demonstrating strong performance even with limited real data.
• The approach outperforms both vanilla active learning and random search, with a significant gap (e.g., 83.03% vs. 71.39% on CIFAR-10) due to the diversity constraint in mixup selection.
• Even when using only synthetic images from out-of-domain data (CIFAR-100), the method achieves 83.03% accuracy with 80K synthetic images, showing robustness to data domain shift.
• The student model trained on mixup images alone (without real data) still achieves 64.10% accuracy with 10K synthetic images, proving the utility of mixup for data augmentation in distillation.
• The blackbox teacher model is more productive than human raters because it can label semantically meaningless mixup images, which humans cannot.