[Paper Review] Rethinking Pre-training and Self-training

Barret Zoph, Golnaz Ghiasi+5 more|arXiv (Cornell University)|Jun 11, 2020

Human Resource Development and Performance Evaluation88 references366 citations

TL;DR

The paper compares pre-training and self-training for vision tasks, showing that strong data augmentation and more unlabeled data diminish pre-training value while self-training consistently provides gains, even when pre-training hurts. It demonstrates self-training's additive benefits and reports new state-of-the-art results on COCO and PASCAL with self-training.

ABSTRACT

Pre-training is a dominant paradigm in computer vision. For example, supervised ImageNet pre-training is commonly used to initialize the backbones of object detection and segmentation models. He et al., however, show a surprising result that ImageNet pre-training has limited impact on COCO object detection. Here we investigate self-training as another method to utilize additional data on the same setup and contrast it against ImageNet pre-training. Our study reveals the generality and flexibility of self-training with three additional insights: 1) stronger data augmentation and more labeled data further diminish the value of pre-training, 2) unlike pre-training, self-training is always helpful when using stronger data augmentation, in both low-data and high-data regimes, and 3) in the case that pre-training is helpful, self-training improves upon pre-training. For example, on the COCO object detection dataset, pre-training benefits when we use one fifth of the labeled data, and hurts accuracy when we use all labeled data. Self-training, on the other hand, shows positive improvements from +1.3 to +3.4AP across all dataset sizes. In other words, self-training works well exactly on the same setup that pre-training does not work (using ImageNet to help COCO). On the PASCAL segmentation dataset, which is a much smaller dataset than COCO, though pre-training does help significantly, self-training improves upon the pre-trained model. On COCO object detection, we achieve 54.3AP, an improvement of +1.5AP over the strongest SpineNet model. On PASCAL segmentation, we achieve 90.5 mIOU, an improvement of +1.5% mIOU over the previous state-of-the-art result by DeepLabv3+.

Motivation & Objective

• Evaluate the effectiveness of ImageNet pre-training for object detection and segmentation under varying data augmentation strengths and labeled data amounts.
• Assess self-training as an alternative to pre-training using unlabeled data from ImageNet and Open Images.
• Compare supervised pre-training, self-supervised pre-training, and self-training to determine their relative benefits and interactions.
• Demonstrate the scalability and flexibility of self-training across architectures, datasets, and tasks (detection and segmentation).

Proposed method

• Systematically vary data augmentation strength with four policies (Augment-S1 to Augment-S4).
• Use EfficientNet-B7 backbone with RetinaNet for object detection on COCO, plus SpineNet variants for state-of-the-art comparisons.
• Apply self-training with a teacher-student framework using pseudo labeling from unlabeled data (ImageNet, Open Images).
• Evaluate both supervised ImageNet pre-training and self-supervised pre-training (SimCLR) as initializations.
• Investigate joint-training and combinations of pre-training, self-training, and joint optimization to assess additive gains.

Experimental results

Research questions

• RQ1 Does ImageNet pre-training help COCO object detection and PASCAL segmentation under varying data augmentation strengths and labeled data sizes?
• RQ2 Is self-training robust and beneficial when pre-training hurts, and how does it interact with augmentation?
• RQ3 How do supervised and self-supervised pre-training compare in their transfer to COCO/segmentation tasks?
• RQ4 Can self-training yield state-of-the-art results across datasets and architectures, and is it additive with pre-training?

Key findings

• Stronger data augmentation and more labeled data reduce or reverse the benefit of pre-training, with strong augmentation sometimes hurting by up to -1.0 AP on COCO.
• Self-training consistently yields gains across data regimes, including when pre-training is detrimental, with +1.3 AP on COCO under strong augmentation.
• Self-training benefits persist across dataset sizes (20%–100%) and are additive with pre-training, e.g., combined gains surpass either method alone.
• Self-supervised pre-training (SimCLR) performs similarly to supervised ImageNet pre-training and also hurts COCO under high-data/strong-augmentation settings, where self-training helps.
• On COCO, self-training with Open Images achieves 54.3 AP, exceeding the previous SpineNet baseline by +1.5 AP; on PASCAL VOC 2012, self-training with NAS-FPN/EfficientNet reaches 90.5 mIOU, +1.5% over prior state-of-the-art.
• Joint-training and combining pre-training, self-training, and joint optimization can yield further improvements, indicating complementary benefits.