[Paper Review] Feature-map-level Online Adversarial Knowledge Distillation

Inseop Chung, Seonguk Park+2 more|arXiv (Cornell University)|Feb 5, 2020

Adversarial Robustness in Machine Learning21 references77 citations

TL;DR

The paper introduces online knowledge distillation that transfers both class-probability knowledge and feature-map distributions via adversarial training, with a cyclic scheme for training multiple networks, yielding improvements especially for small-large network pairs.

ABSTRACT

Feature maps contain rich information about image intensity and spatial correlation. However, previous online knowledge distillation methods only utilize the class probabilities. Thus in this paper, we propose an online knowledge distillation method that transfers not only the knowledge of the class probabilities but also that of the feature map using the adversarial training framework. We train multiple networks simultaneously by employing discriminators to distinguish the feature map distributions of different networks. Each network has its corresponding discriminator which discriminates the feature map from its own as fake while classifying that of the other network as real. By training a network to fool the corresponding discriminator, it can learn the other network's feature map distribution. We show that our method performs better than the conventional direct alignment method such as L1 and is more suitable for online distillation. Also, we propose a novel cyclic learning scheme for training more than two networks together. We have applied our method to various network architectures on the classification task and discovered a significant improvement of performance especially in the case of training a pair of a small network and a large one.

Motivation & Objective

• Motivate improving online knowledge distillation by leveraging intermediate feature maps beyond logits.
• Propose adversarially distilling feature-map distributions between co-trained networks.
• Develop a cyclic learning scheme to efficiently train more than two networks online.
• Demonstrate effectiveness across architectures and scales (including cross-architecture and ImageNet).

Proposed method

• Distill both logit-level knowledge (CE + KL-based mutual distillation) and feature-map-level knowledge (adversarial Distillation).
• Attach a discriminator to each network to distinguish its own feature-map distribution from peers’ distributions.
• Train each network to fool its corresponding discriminator, thereby aligning feature-map distributions.
• Use a transfer layer to handle mismatched feature-map channels when networks differ in architecture.
• Introduce a cyclic learning framework to extend online distillation to more than two networks with reduced discriminators and computation, plus a sequential distillation flow 1→2→…→K→1.

Experimental results

Research questions

• RQ1 Can online distillation benefit from transferring feature-map distributions in addition to logits?
• RQ2 Does adversarial feature-map distillation outperform direct feature-map alignment losses (e.g., L1/L2) in online settings?
• RQ3 Does a cyclic learning scheme effectively scale online distillation to more than two networks?
• RQ4 Do the proposed methods generalize across same- and cross-architecture network pairs and scale to ImageNet?

Key findings

• Adversarial feature-map distillation (AFD) outperforms direct feature-map alignment methods (L1/L1+KD) in online and offline settings.
• AFD provides gains when pairing small and large networks and across same- and cross-architecture pairings.
• Empirical results show notable improvements in CIFAR-100 across multiple architectures and even on ImageNet with DML as a baseline.
• A cyclic learning framework enables efficient online training of three or more networks with competitive or superior performance.
• Ablation studies confirm that both logit-level mutual distillation and adversarial feature-map distillation contribute to performance gains.