[Paper Review] Heterogeneous Contrastive Learning: Encoding Spatial Information for Compact Visual Representations

Xinyue Huo, Lingxi Xie+7 more|arXiv (Cornell University)|Nov 19, 2020

Domain Adaptation and Few-Shot Learning41 references4 citations

TL;DR

This paper proposes Heterogeneous Contrastive Learning (HCL), a method that enhances self-supervised visual representation learning by jointly encoding semantic and spatial features through a dual-branch architecture. By incorporating spatial information via a modified Feature Pyramid Network branch, HCL improves representation efficiency, achieving higher accuracy in instance discrimination and surpassing MoCo-v2 on downstream tasks like object detection and segmentation—while halving pre-training costs.

ABSTRACT

Contrastive learning has achieved great success in self-supervised visual representation learning, but existing approaches mostly ignored spatial information which is often crucial for visual representation. This paper presents heterogeneous contrastive learning (HCL), an effective approach that adds spatial information to the encoding stage to alleviate the learning inconsistency between the contrastive objective and strong data augmentation operations. We demonstrate the effectiveness of HCL by showing that (i) it achieves higher accuracy in instance discrimination and (ii) it surpasses existing pre-training methods in a series of downstream tasks while shrinking the pre-training costs by half. More importantly, we show that our approach achieves higher efficiency in visual representations, and thus delivers a key message to inspire the future research of self-supervised visual representation learning.

Motivation & Objective

• To address the inconsistency between contrastive learning objectives and spatially sensitive data augmentations like cropping and flipping.
• To improve the efficiency and compactness of visual representations by encoding spatial structure alongside semantic features.
• To reduce pre-training costs without sacrificing performance, especially in tasks requiring spatial understanding.
• To demonstrate that heterogeneous feature encoding leads to more efficient and lower-loss representations than homogeneous ones.

Proposed method

• HCL introduces a dual-branch encoder: one branch extracts semantic features from the final average-pooled layer, while a second branch uses a modified Feature Pyramid Network (FPN) to extract spatial features.
• The spatial branch is adapted to output a 14×14 feature map, which is globally pooled to produce a compact vector, ensuring compatibility with the semantic vector in dimensionality.
• The semantic and spatial features are normalized and concatenated before being fed into the contrastive loss function, enabling joint optimization.
• The method is applied only during pre-training; the final model remains compatible with standard downstream tasks without architectural changes.
• The training schedule is split into two stages: first pre-training the semantic branch, then jointly fine-tuning both branches to reduce conflict between learning objectives.
• Horizontal flipping is disabled during training to prevent mismatch between spatial features of flipped and original images, improving downstream performance.

Experimental results

Research questions

• RQ1 Can encoding spatial information during contrastive pre-training improve the quality and efficiency of visual representations?
• RQ2 How does the integration of spatial and semantic features affect performance on downstream tasks like object detection and instance segmentation?
• RQ3 Does HCL reduce the learning inconsistency caused by spatially sensitive data augmentations such as cropping and flipping?
• RQ4 To what extent can pre-training costs be reduced while maintaining or improving performance?
• RQ5 Is the combined use of semantic and spatial features more efficient in terms of representation compactness and instance discrimination accuracy?

Key findings

• HCL achieves a 0.8% higher AP on MS-COCO object detection and 0.7% higher AP on instance segmentation compared to MoCo-v2, despite using only half the pre-training epochs.
• With a 400-epoch total schedule (200 for each stage), HCL outperforms MoCo-v2’s 800-epoch training, demonstrating significant efficiency gains.
• PCA-based compression shows that mixed 128D features (half semantic, half spatial) achieve higher instance discrimination accuracy than purely semantic features of the same dimension.
• Disabling horizontal flipping during training improves downstream performance by 0.4% in detection and 0.5% in LVIS instance segmentation, indicating reduced conflict between spatial and semantic learning.
• The spatial branch’s modified FPN design reduces training cost by ~10% by directly outputting 14×14 features, avoiding additional down-sampling.
• The two-stage training schedule reveals that semantic pre-training is essential; skipping it causes a 0.5% drop in both detection and segmentation APs.