[Paper Review] Dense Semantic Contrast for Self-Supervised Visual Representation Learning

Xiaoni Li, Yu Zhou+6 more|arXiv (Cornell University)|Sep 16, 2021

Domain Adaptation and Few-Shot Learning47 references4 citations

TL;DR

This paper proposes Dense Semantic Contrast (DSC), a self-supervised visual representation learning method that models semantic category decision boundaries at the pixel level by leveraging intra- and inter-image semantic relations. By introducing a multi-granularity contrastive framework that jointly optimizes instance, pixel, and semantic-level representations, DSC achieves state-of-the-art performance on downstream dense prediction tasks such as object detection and semantic segmentation.

ABSTRACT

Self-supervised representation learning for visual pre-training has achieved remarkable success with sample (instance or pixel) discrimination and semantics discovery of instance, whereas there still exists a non-negligible gap between pre-trained model and downstream dense prediction tasks. Concretely, these downstream tasks require more accurate representation, in other words, the pixels from the same object must belong to a shared semantic category, which is lacking in the previous methods. In this work, we present Dense Semantic Contrast (DSC) for modeling semantic category decision boundaries at a dense level to meet the requirement of these tasks. Furthermore, we propose a dense cross-image semantic contrastive learning framework for multi-granularity representation learning. Specially, we explicitly explore the semantic structure of the dataset by mining relations among pixels from different perspectives. For intra-image relation modeling, we discover pixel neighbors from multiple views. And for inter-image relations, we enforce pixel representation from the same semantic class to be more similar than the representation from different classes in one mini-batch. Experimental results show that our DSC model outperforms state-of-the-art methods when transferring to downstream dense prediction tasks, including object detection, semantic segmentation, and instance segmentation. Code will be made available.

Motivation & Objective

• Address the gap between self-supervised pre-training and downstream dense prediction tasks, which require more precise, semantically coherent representations.
• Overcome the limitation of existing instance-level and pixel-level contrastive methods that fail to model semantic category boundaries explicitly.
• Improve representation quality by explicitly mining semantic relations among pixels across images and within images from multiple views.
• Enable multi-granularity learning that integrates low-level (pixel), mid-level (instance), and high-level (semantic category) visual understandings.
• Enhance feature discriminability for dense prediction by aligning representations of pixels from the same semantic class across different images.

Proposed method

• Propose a dense cross-image semantic contrastive learning framework that enforces similarity between pixel representations from the same semantic class within a mini-batch.
• Model intra-image relations by discovering pixel neighbors from multiple augmented views of the same image.
• Use k-nearest neighbors (K-NN) and prototype mining (PM) to identify semantic neighbors across images, forming positive pairs for contrastive learning.
• Integrate multiple granularities—instance, pixel, and semantic—into a unified training objective to jointly optimize for local and global semantic understanding.
• Apply contrastive loss to align representations of pixels sharing the same semantic category while pushing apart representations from different classes.
• Utilize Grad-CAM visualization to analyze and compare feature attention maps, demonstrating improved boundary sensitivity in DSC compared to baselines.

Experimental results

Research questions

• RQ1 Can explicit modeling of semantic category boundaries at the pixel level improve self-supervised representation learning for dense prediction tasks?
• RQ2 How does incorporating multi-granularity representation learning (instance, pixel, and semantic) affect downstream performance?
• RQ3 To what extent does mining inter-image semantic relations improve feature discriminability compared to instance-level or pixel-level contrastive learning alone?
• RQ4 Can a contrastive framework that explicitly models semantic structure outperform methods that rely solely on data augmentation and instance discrimination?
• RQ5 How do different semantic neighbor mining strategies (K-NN vs. prototype mining) impact the final representation quality and convergence stability?

Key findings

• DSC achieves state-of-the-art performance on PASCAL VOC object detection and semantic segmentation, with an AP of 57.1 and mIoU of 57.9 when using all three granularities.
• The ablation study shows that performance improves progressively with each added granularity: instance-only (54.7 AP), pixel+instance (56.4 AP), and full multi-granularity (57.1 AP).
• The DSC model with prototype mining (DSC-PM) achieves better performance than K-NN-based DSC-KM, with a 57.9 mIoU on PASCAL VOC semantic segmentation.
• Grad-CAM visualization confirms that DSC learns more boundary-sensitive features than MoCo-v2 and DenseCL, especially around object edges.
• Models trained with only pixel-level or semantic-level contrastive learning fail to converge, indicating that multi-granularity learning is essential for stability and performance.
• The method effectively reduces misclassification of pixel categories in segmentation tasks, as shown in qualitative comparisons on PASCAL VOC, where DSC produces more accurate and coherent segmentation masks than MoCo-v2 and DenseCL.