[Paper Review] Dense Contrastive Learning for Self-Supervised Visual Pre-Training

Xinlong Wang, Rufeng Zhang+3 more|arXiv (Cornell University)|Nov 18, 2020

Domain Adaptation and Few-Shot Learning48 references40 citations

TL;DR

DenseCL performs dense (pixel-level) contrastive learning with a dense projection head to improve self-supervised pre-training for dense prediction tasks, achieving notable gains over MoCo-v2 and ImageNet supervision with minimal overhead.

ABSTRACT

To date, most existing self-supervised learning methods are designed and optimized for image classification. These pre-trained models can be sub-optimal for dense prediction tasks due to the discrepancy between image-level prediction and pixel-level prediction. To fill this gap, we aim to design an effective, dense self-supervised learning method that directly works at the level of pixels (or local features) by taking into account the correspondence between local features. We present dense contrastive learning, which implements self-supervised learning by optimizing a pairwise contrastive (dis)similarity loss at the pixel level between two views of input images. Compared to the baseline method MoCo-v2, our method introduces negligible computation overhead (only <1% slower), but demonstrates consistently superior performance when transferring to downstream dense prediction tasks including object detection, semantic segmentation and instance segmentation; and outperforms the state-of-the-art methods by a large margin. Specifically, over the strong MoCo-v2 baseline, our method achieves significant improvements of 2.0% AP on PASCAL VOC object detection, 1.1% AP on COCO object detection, 0.9% AP on COCO instance segmentation, 3.0% mIoU on PASCAL VOC semantic segmentation and 1.8% mIoU on Cityscapes semantic segmentation. Code is available at: https://git.io/AdelaiDet

Motivation & Objective

• Bridge the gap between self-supervised pre-training and dense prediction tasks (e.g., object detection, semantic segmentation).
• Develop a dense, pixel-level contrastive learning framework that preserves spatial information.
• Introduce a dense projection head and a dense contrastive loss to match local features across views.
• Demonstrate that dense pre-training yields substantial improvements across multiple downstream dense prediction benchmarks.

Proposed method

• Introduce a dense projection head that outputs dense feature maps instead of a single global vector.
• Compute a joint loss combining global and dense contrastive terms with a balanced weight (λ = 0.5).
• Define a dense contrastive loss (InfoNCE-like) over SxS local features extracted from dense projection outputs.
• Establish dense correspondences across views by matching backbone feature maps, enabling correct positive key pairing for local features.
• Train with a fully-convolutional pipeline to preserve pixel-level correspondence and reduce overhead.
• Keep similar parameter counts to MoCo-v2 and show negligible additional computation (<1%).

Experimental results

Research questions

• RQ1 Can dense, pixel-level contrastive learning improve transfer to dense prediction tasks beyond image-level objectives?
• RQ2 How should the dense projection head and dense correspondence be designed to maximize performance with minimal overhead?
• RQ3 What is the impact of the balance between global and dense losses (λ) and the grid size (S) on downstream performance?
• RQ4 Does learning dense local features improve object detection, instance segmentation, and semantic segmentation across diverse datasets?

Key findings

• DenseCL consistently improves over MoCo-v2 on downstream dense tasks (e.g., +2.0 AP for VOC object detection, +1.1 AP for COCO object detection, +0.9 AP for COCO instance segmentation).
• DenseCL yields substantial semantic segmentation gains (e.g., +3.0 mIoU on VOC and +1.8 mIoU on Cityscapes).
• DenseCL pre-trained on COCO with 800 epochs achieves 57.2 AP on VOC (vs 57.0 AP for ImageNet MoCo-v2), and 1600-epoch COCO pre-training surpasses MoCo-v2 IN by 0.2 AP in some settings.
• DenseCL achieves top performance in VOC/Coco benchmarks even when pre-trained on COCO, often matching or surpassing supervised ImageNet pre-training (e.g., VOC mIoU 67.5 with COCO DenseCL vs 67.7 with supervised ImageNet).
• Dense correspondence extraction using backbone features (F1, F2) yields best results for dense matching (Table 6).
• DenseCL introduces negligible overhead (<1%) in training time per epoch compared to MoCo-v2.