[论文解读] Are all negatives created equal in contrastive instance discrimination?

Tiffany Cai, Jonathan Frankle+2 more|arXiv (Cornell University)|Oct 13, 2020

Categorization, perception, and language参考文献 31被引用 57

一句话总结

本论文在 MoCo v2 的对比实例判别 CID 中，只有最困难的 5% 负样本是必要且足以实现接近全部下游准确率，而最容易的 95% 不必要；极其困难的 0.1% 在某些设置下甚至有害。

ABSTRACT

Self-supervised learning has recently begun to rival supervised learning on computer vision tasks. Many of the recent approaches have been based on contrastive instance discrimination (CID), in which the network is trained to recognize two augmented versions of the same instance (a query and positive) while discriminating against a pool of other instances (negatives). The learned representation is then used on downstream tasks such as image classification. Using methodology from MoCo v2 (Chen et al., 2020), we divided negatives by their difficulty for a given query and studied which difficulty ranges were most important for learning useful representations. We found a minority of negatives -- the hardest 5% -- were both necessary and sufficient for the downstream task to reach nearly full accuracy. Conversely, the easiest 95% of negatives were unnecessary and insufficient. Moreover, the very hardest 0.1% of negatives were unnecessary and sometimes detrimental. Finally, we studied the properties of negatives that affect their hardness, and found that hard negatives were more semantically similar to the query, and that some negatives were more consistently easy or hard than we would expect by chance. Together, our results indicate that negatives vary in importance and that CID may benefit from more intelligent negative treatment.

研究动机与目标

• 激发对对比实例判别 (CID) 中负样本相对重要性的理解。
• 量化不同难度的负样本对下游 ImageNet 线性准确度的贡献。
• 识别区分困难与容易负样本的语义特征。
• 探讨是否有某些负样本在不同查询中持续影响学习。
• 提出对 CID 中更智能负样本采样的启示。

提出的方法

• 使用带有 ResNet-50 编码器和 MLP 投影头的 MoCo v2。
• 将负样本难度定义为查询和负样本在对比空间中的归一化嵌入的点积。
• 通过移除子集来评估负样本的必要性和充分性，并在 ImageNet 线性分类上测量下游准确度。
• 在两个温度（0.07 和 0.20）以及三个随机种子下进行评估。
• 通过类别标签和基于 WordNet 的相似性度量分析负样本的语义相似性。

实验结果

研究问题

• RQ1 在 CID 中，哪些难度等级的负样本对实现高下游准确度是必要的？
• RQ2 单独用于预训练时，最困难的负样本是否足够？
• RQ3 极端困难的负样本是否会损害学习，如若如此，原因何在？
• RQ4 哪些语义属性区分易与难的负样本？
• RQ5 研究结果能否为 CID 的课程化或选择性负样本采样提供借鉴？

主要发现

• 最容易的 95% 负样本对于达到完全精确度而言既不必要也不足够；前 5% 最困难的负样本是必要且充分的。
• 仅用最困难的 5% 负样本进行训练，在基线 Top-1 精度的差距不超过 0.7 个点；而训练最容易的 95% 会降低性能。
• 在较低温度下，极其困难的 0.1% 负样本有害，部分有益于移除，尤其是同一类别的负样本。
• 困难负样本往往在语义上更接近查询；一些易负样本则呈负相关但在语义上与查询相似。
• 存在在不同查询中始终较难或较易的负样本，暗示在队列中维持持续困难负样本可能带来收益。