[论文解读] Probing Representation Forgetting in Supervised and Unsupervised Continual Learning

MohammadReza Davari, Nader Asadi+3 more|arXiv (Cornell University)|Mar 24, 2022

Domain Adaptation and Few-Shot Learning被引用 4

一句话总结

本文提出了一种新颖的持续学习（CL）评估框架，采用最优线性探测器（LP）衡量表征遗忘，结果表明：使用对比学习（SupCon）进行的朴素微调导致的表征遗忘极少——通常优于专门设计的CL方法。其主要贡献在于证明：基于SupCon的微调配合小规模基于记忆的原型召回机制，可在无需显式遗忘控制的情况下实现快速恢复与优异性能。

ABSTRACT

Continual Learning research typically focuses on tackling the phenomenon of catastrophic forgetting in neural networks. Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model when the task, or more broadly the data distribution, being trained on changes. In supervised learning problems this forgetting, resulting from a change in the model's representation, is typically measured or observed by evaluating the decrease in old task performance. However, a model's representation can change without losing knowledge about prior tasks. In this work we consider the concept of representation forgetting, observed by using the difference in performance of an optimal linear classifier before and after a new task is introduced. Using this tool we revisit a number of standard continual learning benchmarks and observe that, through this lens, model representations trained without any explicit control for forgetting often experience small representation forgetting and can sometimes be comparable to methods which explicitly control for forgetting, especially in longer task sequences. We also show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning. Based on our results, we show that a simple yet competitive approach is to learn representations continually with standard supervised contrastive learning while constructing prototypes of class samples when queried on old samples.

研究动机与目标

• 为解决标准准确率指标在衡量灾难性遗忘时的局限性，该指标可能无法区分微小表征变化与真实知识损失。
• 探究在标准微调和自监督CL方法中，是否发生表征遗忘（即先前任务有用特征的丢失）。
• 通过基于线性探测的分析，评估模型容量和损失函数对表征遗忘的影响。
• 提出一种简单且内存高效的持续学习快速记忆方法，避免使用显式遗忘控制机制。
• 证明表征效用（而不仅仅是任务准确率）是评估持续学习性能的关键指标。

提出的方法

• 使用在神经网络中间层特征激活上训练的最优线性探测器（LP），评估先前任务表征的有用性。
• 将表征遗忘定义为在新任务上训练后，LP分类器在旧任务数据上的性能下降。
• 在监督和自监督设置下，将该评估方法应用于标准CL基准（如SplitCIFAR100、ImageNet-Scenes-CUB-Flowers）。
• 提出一种结合SupCon微调与小规模类别原型记忆（NME）的方法，以实现在旧任务上的快速恢复。
• 将基于LP的遗忘度量与标准准确率指标进行比较，并使用CKA（中心化核对齐）作为补充的相似性度量。
• 在多个任务序列中，使用标准ResNet和VGG架构分析深度方向上的表征变化。

Figure 1 : Performance on ImageNet during the sequence (ImageNet→Scenes→CUB→Flowers) using ResNet18. We observe that although observed accuracy heavily degrades the LP accuracy, in finetuning does not decay as drastically and can rival LP accuracy of methods such as LwF and EWC. Moreover, we observe

实验结果

研究问题

• RQ1 标准准确率在多大程度上无法捕捉持续学习中的真实表征遗忘？
• RQ2 在不同模型架构和损失函数下，表征遗忘如何变化？
• RQ3 使用SupCon或SimCLR进行的朴素微调能否实现优于专用CL方法的表征保留？
• RQ4 增加模型深度或宽度是否能减少表征遗忘？这种效应是否仅在LP评估中可见？
• RQ5 仅使用SupCon与小规模原型记忆的简单方法，能否在无需显式遗忘控制的情况下实现快速记忆？

主要发现

• 在2任务SplitCIFAR10设置下，朴素微调的表征遗忘显著低于标准准确率指标所提示的水平：ResNet最后一层块的LP准确率仅下降5.72%。
• 采用NME记忆的SupCon微调在10任务SplitCIFAR100序列上实现了53.9%的平均准确率，优于LwF（47.7%），并接近ER-M5（54.8%），且内存使用量相当。
• 早期网络层的表征效用保持较高：在ResNet的B-0块中，LP准确率在任务2后提升了1.08%，B-1块中提升了1.26%，表明低级特征遗忘极少。
• 遗忘主要集中在深层：ResNet B-5块的LP准确率下降5.72%，但远低于观察到的任务准确率22%的下降幅度。
• 通过LP评估，更宽或更深的模型表现出更少的表征遗忘，而这一趋势在标准准确率指标中并不明显。
• 所提出的SupCon + NME方法可在无需训练时访问记忆的情况下实现旧任务的快速恢复，展现出强大的实际应用价值。

Figure 3 : 10-Task SplitCIFAR100. We compare observed acc. and linear probe acc. Naively finetuning with CE does poorly if using the observed accuracy. However using the LP based evaluation we observe that performance gap to other methods is lower. Furthermore when finetuning is performed instead wi