[论文解读] Multi-Task Self-Training for Learning General Representations

Golnaz Ghiasi, Barret Zoph+3 more|arXiv (Cornell University)|Aug 25, 2021

Advanced Neural Network Applications参考文献 63被引用 5

一句话总结

该论文提出多任务自训练（MuST），一种通过共享的未标注数据集生成的伪标签，将独立训练的专用教师模型的知识蒸馏到单一通用学生模型中的方法。MuST在多个视觉任务上实现了最先进性能，尤其在大规模数据上训练时，优于专用监督模型和对比学习方法（如SimCLR）。

ABSTRACT

Despite the fast progress in training specialized models for various tasks, learning a single general model that works well for many tasks is still challenging for computer vision. Here we introduce multi-task self-training (MuST), which harnesses the knowledge in independent specialized teacher models (e.g., ImageNet model on classification) to train a single general student model. Our approach has three steps. First, we train specialized teachers independently on labeled datasets. We then use the specialized teachers to label an unlabeled dataset to create a multi-task pseudo labeled dataset. Finally, the dataset, which now contains pseudo labels from teacher models trained on different datasets/tasks, is then used to train a student model with multi-task learning. We evaluate the feature representations of the student model on 6 vision tasks including image recognition (classification, detection, segmentation)and 3D geometry estimation (depth and surface normal estimation). MuST is scalable with unlabeled or partially labeled datasets and outperforms both specialized supervised models and self-supervised models when training on large scale datasets. Lastly, we show MuST can improve upon already strong checkpoints trained with billions of examples. The results suggest self-training is a promising direction to aggregate labeled and unlabeled training data for learning general feature representations.

研究动机与目标

• 为解决学习一个在多种任务上表现良好的单一通用视觉模型的挑战。
• 克服大规模多任务数据集在计算机视觉中数据稀缺和标注负担过重的问题。
• 探究来自专用教师模型的伪标签是否能有效将知识迁移至统一的学生模型。
• 评估多任务自训练是否能产生优于监督或自监督预训练的通用表征。

提出的方法

• 在特定任务的标注数据集上独立训练专用教师模型（例如，ImageNet用于分类，COCO用于检测，MiDaS用于深度估计）。
• 利用训练好的教师模型在大规模共享未标注数据集（如ImageNet）上生成多任务伪标签。
• 使用伪标签数据集，通过端到端的多任务学习训练单一通用学生模型，采用共享的ResNet-FPN主干网络和任务特定的分类头。
• 在学生模型训练过程中应用尺度抖动和标准训练协议，不使用数据增强或标签平滑。
• 通过网格搜索超参数对模型在下游任务上进行微调以最大化性能。
• 采用简单的模型架构，通过任务间共享特征，高效学习通用表征。

Figure 1: An overview of Multi-Task Self-Training (MuST). Specialized Teacher represents a supervised model trained on a single task and dataset ( e.g . , classification model trained on ImageNet). Specialized Teacher models are trained independently on their own tasks and datasets. They then genera

实验结果

研究问题

• RQ1 通过利用多个专用教师模型生成的伪标签，单一学生模型能否学习到通用视觉表征？
• RQ2 多任务自训练是否在下游视觉任务上优于监督和自监督预训练？
• RQ3 MuST能否在大规模或部分标注的数据集上有效扩展以提升泛化能力？
• RQ4 在未直接进行多任务标注的情况下，来自多样化教师模型的知识是否可迁移至统一的学生模型？

主要发现

• MuST学生模型在六项视觉任务（包括分类、检测、分割、深度估计和表面法线预测）上的性能与专用教师模型相当或更优。
• 在PASCAL VOC分割基准上，MuST学生模型达到53.8 mIoU，优于监督基线，并与专用语义分割教师模型性能相当。
• 在NYU Depth v2数据集上，MuST学生模型在相对误差比为1.25内的深度精度达到91.2%，超过SimCLR，并与专用MiDaS教师模型性能持平。
• 在DIODE数据集上进行表面法线估计时，MuST学生模型在角度误差小于11.25°时达到87.4%的准确率，超过SimCLR和监督基线。
• MuST学生模型在使用数十亿样本预训练的ImageNet检查点基础上进一步提升，证明其在大规模数据上的可扩展性和有效性。
• 即使仅使用教师模型生成的伪标签进行训练，学生模型仍能保留专用教师模型的大部分迁移性能，验证了伪标签蒸馏的有效性。

Figure 2: Examples of pseudo labels on ImageNet. Left: bounding boxes labeled with an Objects365 teacher model. Middle: semantic segmentation labeled with a COCO teacher model. Right: depth labeled with a MiDaS teacher model.