[논문 리뷰] Knowledge Distillation via Route Constrained Optimization

Jin Xiao, Baoyun Peng외 6명|arXiv (Cornell University)|2019. 04. 19.

Advanced Neural Network Applications참고 문헌 34인용 수 36

한 줄 요약

RCO는 교사의 훈련 경로를 따라 쉬움에서 어려움 순서의 교사 체크포인트를 따라 학생을 학습시키며, CIFAR-100, ImageNet, MegaFace에서 표준 지식 증류보다 성능을 향상시킨다. 이는 앵커 포인트 가이드를 사용해 일치성 격차를 줄인다.

ABSTRACT

Distillation-based learning boosts the performance of the miniaturized neural network based on the hypothesis that the representation of a teacher model can be used as structured and relatively weak supervision, and thus would be easily learned by a miniaturized model. However, we find that the representation of a converged heavy model is still a strong constraint for training a small student model, which leads to a high lower bound of congruence loss. In this work, inspired by curriculum learning we consider the knowledge distillation from the perspective of curriculum learning by routing. Instead of supervising the student model with a converged teacher model, we supervised it with some anchor points selected from the route in parameter space that the teacher model passed by, as we called route constrained optimization (RCO). We experimentally demonstrate this simple operation greatly reduces the lower bound of congruence loss for knowledge distillation, hint and mimicking learning. On close-set classification tasks like CIFAR100 and ImageNet, RCO improves knowledge distillation by 2.14% and 1.5% respectively. For the sake of evaluating the generalization, we also test RCO on the open-set face recognition task MegaFace.

연구 동기 및 목표

• 교사의 최적화 경로 관점에서 지식 증류를 재고한다.
• 교사의 궤적을 따라 앵커 포인트를 통해 학생을 안내하는 Route Constrained Optimization(RCO)을 제안한다.
• RCO가 분류 및 얼굴 인식 과제에서 표준 KD 및 최첨단 방법들을 능가한다는 것을 보여준다.

제안 방법

• 교사(φ_t)와 학생(φ_s)를 정의하고 소프트맥스 출력과 KL/MSE 손실을 사용해 지식을 증류한다.
• 수렴된 교사로 감독하는 것은 용량 차이로 인해 학생에게 더 어려울 수 있음을 보인다.
• 교사의 학습 경로를 따라 앵커 포인트 C_1,...,C_n을 도입하고 각 단계에서 φ_t(·;W_{C_i})를 모사하도록 학생을 순차적으로 학습시킨다.
• 두 가지 앵커 포인트 선택 전략을 제공한다: Equal Epoch Interval(EEI)과 KL 발산에 기반해 정보량 많은 앵커를 선택하는 Greedy Search(GS).
• 단계 i에서의 Loss_i를 Loss_i = H(y, φ_s(x;W_s)) + λ H(φ_s(x;W_s), φ_t(x;W_{C_i}))로 형식화한다.
• 교과학습 률과 교사의 궤적이 만들려는 완화-강화 순서를 논의한다.

실험 결과

연구 질문

• RQ1 Can supervision from intermediate states along the teacher’s training path reduce the congruence loss for a small student compared to using only a final converged teacher?
• RQ2 Do anchor-point selection strategies (EEI, GS) effectively create a curriculum that improves student performance and training efficiency?
• RQ3 Are the gains from RCO consistent across CIFAR-100, ImageNet-1K, and MegaFace face recognition tasks?
• RQ4 Can RCO be combined with existing knowledge transfer methods to boost their performance?

주요 결과

Experiment	Model	Top-1	Top-5
CIFAR-100 (KD vs RCO)	S-KD (MobileNetV2)	68.71	-
CIFAR-100 (RCO)	S-RCO (MobileNetV2)	70.85	-
ImageNet (Student)	Student-KD (MobileNetV2)	66.75	87.30
ImageNet (RCO)	Student-RCO (MobileNetV2)	68.21	88.04

• RCO improves KD by about 2.14% top-1 on CIFAR-100 and 1.5% top-1 on ImageNet-1K over standard KD.
• On MegaFace face recognition, RCO achieves 84.3% accuracy with only 0.8M parameters, surpassing prior methods.
• RCO consistently outperforms KD and other SOTA methods across CIFAR-100, ImageNet, and MegaFace.
• Greedy Search anchor strategy yields the best performance among evaluated strategies, with EEI and GS offering trade-offs between training time and accuracy.
• RCO can be combined with previous knowledge transfer methods to further boost performance.