Skip to main content
Nubint
QUICK REVIEW

[论文解读] Learning Metrics from Teachers: Compact Networks for Image Embedding

Lu Yu, Vacit Oguz Yazici|arXiv (Cornell University)|Apr 7, 2019
Face recognition and analysis参考文献 38被引用 8
一句话总结

本文提出从大型教师网络到紧凑学生网络的知识蒸馏方法，以实现高效的图像嵌入学习。通过引入两种新型损失函数——绝对度量蒸馏与相对度量蒸馏，使如MobileNet-0.25等小型网络在CUB-200-2011数据集上的检索性能显著提升，Recall@1从27.5%提高至44.6%。

ABSTRACT

Metric learning networks are used to compute image embeddings, which are widely used in many applications such as image retrieval and face recognition. In this paper, we propose to use network distillation to efficiently compute image embeddings with small networks. Network distillation has been successfully applied to improve image classification, but has hardly been explored for metric learning. To do so, we propose two new loss functions that model the communication of a deep teacher network to a small student network. We evaluate our system in several datasets, including CUB-200-2011, Cars-196, Stanford Online Products and show that embeddings computed using small student networks perform significantly better than those computed using standard networks of similar size. Results on a very compact network (MobileNet-0.25), which can be used on mobile devices, show that the proposed method can greatly improve Recall@1 results from 27.5\% to 44.6\%. Furthermore, we investigate various aspects of distillation for embeddings, including hint and attention layers, semi-supervised learning and cross quality distillation. (Code is available at https://github.com/yulu0724/EmbeddingDistillation.)

研究动机与目标

  • 解决在计算资源和电池容量有限的移动设备上部署大型且高精度度量学习网络的挑战。
  • 在不增加模型大小的前提下，提升小型学生网络在图像嵌入任务中的性能。
  • 探索知识蒸馏在度量学习中的应用，该领域相较于分类任务仍属研究不足。
  • 研究跨质量蒸馏，实现从高质量图像输入向低质量图像输入的知识迁移。
  • 评估不同蒸馏组件的有效性，包括提示层、注意力蒸馏以及半监督学习。

提出的方法

  • 提出一种相对教师损失，指导学生网络复现图像对之间的相对距离，而非精确嵌入。
  • 引入一种绝对教师损失，引导学生网络直接匹配教师网络的输出嵌入。
  • 在教师和学生网络的训练过程中，以三元组损失作为主要度量学习目标。
  • 将知识蒸馏损失（绝对与相对）与三元组损失结合，指导学生网络的训练。
  • 通过提示层和注意力图，将教师网络的中间特征表示迁移至学生网络。
  • 通过在蒸馏过程中引入未标注数据，实现半监督训练，以提升泛化能力。
Figure 1: Graphical illustration of the two knowledge distillation losses we propose for metric learning. $L_{KD}^{abs}$ aims to minimize the distance between the student and teacher embedding of the same image. $L_{KD}^{rel}$ compares the distance in the embedding of the teacher between two images,
Figure 1: Graphical illustration of the two knowledge distillation losses we propose for metric learning. $L_{KD}^{abs}$ aims to minimize the distance between the student and teacher embedding of the same image. $L_{KD}^{rel}$ compares the distance in the embedding of the teacher between two images,

实验结果

研究问题

  • RQ1知识蒸馏能否有效将大型教师网络的度量学习能力迁移至小型学生网络？
  • RQ2在度量蒸馏中，相对教师（教导成对距离）是否优于绝对教师（教导精确嵌入）？
  • RQ3当学生网络在低质量图像上训练而教师网络使用高质量图像时，蒸馏的效果如何？
  • RQ4在蒸馏过程中使用未标注数据对嵌入质量有何影响？
  • RQ5通过注意力或提示层进行蒸馏是否能进一步提升紧凑网络的性能？

主要发现

  • 相对教师损失始终优于绝对教师损失，在使用MobileNet-0.25作为学生网络时，CUB-200-2011数据集上的Recall@1提高了17.1%。
  • 以MobileNet-0.25为学生网络，蒸馏后Recall@1从27.5%提升至44.6%，表明在移动端优化架构上具有强大性能。
  • 在Cars-196数据集上，与未使用同尺寸教师网络训练的学生网络相比，蒸馏使Recall@1提升了29.9%。
  • 跨质量蒸馏在低分辨率及非定位退化设置下显著提升了性能，其中相对教师在Cars-196数据集上对退化输入实现了61.9%的Recall@1。
  • 结合未标注数据的半监督蒸馏进一步提升了性能，表明利用大规模弱监督数据具有显著优势。
  • 在蒸馏中使用注意力与提示层有助于改善特征迁移，尤其在具有挑战性的视觉领域中表现突出。
Figure 2: Illustration of difference between absolute and relative teacher. (left) Example of four data points in the embedding space of teacher. We consider two samples from two classes (indicated by square and star). (middle and right) show the absolute and relative loss for two student embeddings
Figure 2: Illustration of difference between absolute and relative teacher. (left) Example of four data points in the embedding space of teacher. We consider two samples from two classes (indicated by square and star). (middle and right) show the absolute and relative loss for two student embeddings

更好的研究，从现在开始

从阅读论文到最终审阅，大幅缩短您的研究时间。

无需绑定信用卡

本解读由 AI 生成，并经人工编辑审核。