[Paper Review] Optimizing YOLOv5s Object Detection through Knowledge Distillation algorithm

Guanming Huang, Aoran Shen+4 more|arXiv (Cornell University)|Oct 16, 2024

Industrial Vision Systems and Defect Detection4 citations

TL;DR

This paper proposes knowledge distillation using YOLOv5l as a teacher to optimize YOLOv5s as a student model, demonstrating that increasing distillation temperature enhances detection accuracy. At an optimal temperature, the student model achieves superior mAP50 and mAP50-95 scores compared to the original YOLOv5s, with stable convergence after 150 training epochs.

ABSTRACT

This paper explores the application of knowledge distillation technology in target detection tasks, especially the impact of different distillation temperatures on the performance of student models. By using YOLOv5l as the teacher network and a smaller YOLOv5s as the student network, we found that with the increase of distillation temperature, the student's detection accuracy gradually improved, and finally achieved mAP50 and mAP50-95 indicators that were better than the original YOLOv5s model at a specific temperature. Experimental results show that appropriate knowledge distillation strategies can not only improve the accuracy of the model but also help improve the reliability and stability of the model in practical applications. This paper also records in detail the accuracy curve and loss function descent curve during the model training process and shows that the model converges to a stable state after 150 training cycles. These findings provide a theoretical basis and technical reference for further optimizing target detection algorithms.

Motivation & Objective

• To investigate the impact of knowledge distillation on YOLOv5s object detection performance.
• To analyze how varying distillation temperature affects student model accuracy and stability.
• To improve the detection accuracy of YOLOv5s without increasing model complexity.
• To provide a practical, effective method for enhancing lightweight object detectors in real-world applications.

Proposed method

• Employ YOLOv5l as the teacher network to guide knowledge distillation into a smaller YOLOv5s student model.
• Apply soft label distillation with temperature scaling to transfer knowledge from teacher to student.
• Train the student model using a combined loss function that includes both ground-truth and distillation losses.
• Systematically vary the distillation temperature to evaluate its effect on model performance.
• Monitor training dynamics via accuracy and loss curves to assess convergence and stability.
• Train the model for 150 epochs to ensure convergence and reliable performance evaluation.

Experimental results

Research questions

• RQ1 How does varying the distillation temperature affect the mAP50 and mAP50-95 performance of the YOLOv5s student model?
• RQ2 Can knowledge distillation from YOLOv5l improve the accuracy of YOLOv5s beyond its original performance?
• RQ3 At what distillation temperature does the student model achieve optimal detection accuracy?
• RQ4 How stable and reliable is the student model's performance after knowledge distillation?
• RQ5 Does the training process converge to a stable state, and after how many epochs?

Key findings

• The student model achieved higher mAP50 and mAP50-95 scores than the original YOLOv5s at an optimal distillation temperature.
• Detection accuracy increased with rising distillation temperature, peaking at a specific value before plateauing or degrading.
• The model converged to a stable state after 150 training epochs, as indicated by smooth loss and accuracy curves.
• Knowledge distillation enhanced both the accuracy and reliability of the lightweight YOLOv5s model.
• The use of soft labels with temperature scaling significantly improved knowledge transfer from YOLOv5l to YOLOv5s.