[Paper Review] Training Shallow and Thin Networks for Acceleration via Knowledge Distillation with Conditional Adversarial Networks

Zheng Xu, Yen-Chang Hsu+1 more|arXiv (Cornell University)|Sep 1, 2017

Anomaly Detection Techniques and ApplicationsComputer Science83 citations

TL;DR

The paper introduces a GAN-based learning loss for knowledge distillation to train shallow, thin student networks from a deep, wide teacher, improving acceleration-ready models without large accuracy loss. It shows the GAN approach outperforms traditional KD across CIFAR-10/100 and ImageNet32, especially for smaller students.

ABSTRACT

There is an increasing interest on accelerating neural networks for real-time applications. We study the student-teacher strategy, in which a small and fast student network is trained with the auxiliary information learned from a large and accurate teacher network. We propose to use conditional adversarial networks to learn the loss function to transfer knowledge from teacher to student. The proposed method is particularly effective for relatively small student networks. Moreover, experimental results show the effect of network size when the modern networks are used as student. We empirically study the trade-off between inference time and classification accuracy, and provide suggestions on choosing a proper student network.

Motivation & Objective

• Motivate network acceleration by training lightweight student models via knowledge transfer from a strong teacher.
• Propose a conditional adversarial network-based loss to transfer dark knowledge beyond fixed KL-divergence targets.
• Demonstrate that the learned loss is particularly beneficial for shallow/thin students and modern residual architectures.
• Analyze how student capacity, depth/width, and inference-time trade-offs affect performance.

Proposed method

• Use a deep, wide WRN as a fixed teacher and train a shallow, thin WRN student.
• Replace fixed KD loss with a GAN-based learned loss where a discriminator distinguishes teacher vs. student logits, guiding the student to produce teacher-like distributions.
• Augment the GAN loss with an auxiliary classifier component to provide category-level alignment and an L1 instance-level alignment term between student logits and teacher logits (L1 loss).
• Combine the learned GAN-based loss with the supervised cross-entropy loss for training the student (no manual tuning of loss weights).
• Experiment with a 3-layer MLP discriminator and analyze robustness to discriminator depth, training stability, and multi-task signals.

Experimental results

Research questions

• RQ1 Can a GAN-based learned loss transfer dark knowledge more effectively than fixed KD losses for small, fast student networks?
• RQ2 How does student capacity (depth/width) affect knowledge transfer effectiveness and acceleration potential?
• RQ3 What are the trade-offs between inference time, parameter count, and accuracy when using GAN-based distillation on modern residual architectures?
• RQ4 Is the learned loss robust to discriminator architecture and training dynamics across datasets of varying scale (CIFAR-10/100, ImageNet32)?

Key findings

• GAN-based knowledge transfer improves small networks across CIFAR-10, CIFAR-100, and ImageNet32 compared with standard KD.
• For CIFAR-10/100, the GAN method reduces error relative to the baseline student and outperforms fixed KD (across tested temperatures).
• On ImageNet32, the GAN approach enables a smaller student to approach teacher performance more closely than KD does.
• The benefit of the GAN-based approach is most pronounced for shallow/thin students, with effectiveness sustained as student capacity grows.
• Increasing depth of the discriminator has limited impact and the 3-layer MLP is sufficient and stable for these experiments.