[Paper Review] Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning

Yifan Zhang, Bryan Hooi+3 more|arXiv (Cornell University)|Feb 12, 2021

Domain Adaptation and Few-Shot LearningComputer Science78 references29 citations

TL;DR

The paper introduces Core-tuning, a contrast-regularized fine-tuning method for contrastive self-supervised visual models that uses hard-pair mining and a focal contrastive loss to improve downstream classification and segmentation performance.

ABSTRACT

Contrastive self-supervised learning (CSL) has attracted increasing attention for model pre-training via unlabeled data. The resulted CSL models provide instance-discriminative visual features that are uniformly scattered in the feature space. During deployment, the common practice is to directly fine-tune CSL models with cross-entropy, which however may not be the best strategy in practice. Although cross-entropy tends to separate inter-class features, the resulting models still have limited capability for reducing intra-class feature scattering that exists in CSL models. In this paper, we investigate whether applying contrastive learning to fine-tuning would bring further benefits, and analytically find that optimizing the contrastive loss benefits both discriminative representation learning and model optimization during fine-tuning. Inspired by these findings, we propose Contrast-regularized tuning (Core-tuning), a new approach for fine-tuning CSL models. Instead of simply adding the contrastive loss to the objective of fine-tuning, Core-tuning further applies a novel hard pair mining strategy for more effective contrastive fine-tuning, as well as smoothing the decision boundary to better exploit the learned discriminative feature space. Extensive experiments on image classification and semantic segmentation verify the effectiveness of Core-tuning.

Motivation & Objective

• Motivation: improve fine-tuning of contrastive self-supervised learning (CSL) models to better exploit their discriminative feature space.
• Demonstrate that contrastive loss during fine-tuning provides regularization and optimization benefits.
• Develop Core-tuning with hard sample mining and smooth classifier learning to enhance downstream performance.
• Show effectiveness across image classification and semantic segmentation, plus domain generalization and robustness aspects.

Proposed method

• Theoretical analysis showing how contrastive loss regularizes representation learning and optimizes fine-tuning (Theorems 1 and 2).
• Introduction of Core-tuning with hardness-directed mixup to create hard positive and hard negative pairs for each anchor.
• Use of a focal contrastive loss to give higher weight to hard positive pairs during fine-tuning.
• Projection head G_c to obtain normalized contrastive features for reweighting the loss.
• Combination of cross-entropy and focal contrastive loss with a Mixup-based data augmentation to learn a smoother classifier.
• Training objective: minimize L_ce^m + eta * L_con^f, plus mixing-based classifier training to enhance generalization.

Experimental results

Research questions

• RQ1 Can applying contrastive learning to fine-tuning CSL models improve downstream performance compared to standard cross-entropy fine-tuning?
• RQ2 How can hard sample mining and classifier smoothing be integrated to maximize the benefits of contrastive fine-tuning?
• RQ3 Does Core-tuning improve domain generalization and adversarial robustness for CSL models?
• RQ4 Is the proposed approach generalizable across architectures, pre-training methods, and downstream tasks such as semantic segmentation?

Key findings

• Core-tuning significantly improves fine-tuning performance over CE-tuning and other baselines across multiple datasets.
• Ablation shows each component (hard pair mining, focal loss, mixup-based mixing, and smooth classifier learning) contributes to gains.
• Core-tuning yields higher average top-1 accuracy on 9 natural image datasets with MoCo-v2 Pre-trained ResNet-50, outperforming CE-tuning and several baselines.
• The method also improves semantic segmentation performance on PASCAL VOC when fine-tuned from CSL pre-trained backbones.
• Core-tuning demonstrates better cross-domain generalization on PACS, VLCS, and Office-Home datasets and shows robustness in adversarial settings (adversarial training results mentioned).