[Paper Review] Unsupervised Representation Learning by InvariancePropagation

Feng Wang, Huaping Liu+2 more|arXiv (Cornell University)|Oct 7, 2020

Domain Adaptation and Few-Shot Learning41 references21 citations

TL;DR

This paper proposes Invariance Propagation (InvP), a novel unsupervised representation learning method that discovers semantically consistent, category-level positive samples via recursive k-nearest neighbor propagation in representation space. By focusing on hard positive samples with high intra-class variation, InvP achieves state-of-the-art performance, attaining 71.3% top-1 accuracy on ImageNet linear classification and 78.2% top-5 accuracy with only 1% labeled data fine-tuning.

ABSTRACT

Unsupervised learning methods based on contrastive learning have drawn increasing attention and achieved promising results. Most of them aim to learn representations invariant to instance-level variations, which are provided by different views of the same instance. In this paper, we propose Invariance Propagation to focus on learning representations invariant to category-level variations, which are provided by different instances from the same category. Our method recursively discovers semantically consistent samples residing in the same high-density regions in representation space. We demonstrate a hard sampling strategy to concentrate on maximizing the agreement between the anchor sample and its hard positive samples, which provide more intra-class variations to help capture more abstract invariance. As a result, with a ResNet-50 as the backbone, our method achieves 71.3% top-1 accuracy on ImageNet linear classification and 78.2% top-5 accuracy fine-tuning on only 1% labels, surpassing previous results. We also achieve state-of-the-art performance on other downstream tasks, including linear classification on Places205 and Pascal VOC, and transfer learning on small scale datasets.

Motivation & Objective

• To address the limitation of contrastive learning methods that focus on instance-level invariance by instead learning category-level invariance through inter-instance relations.
• To improve representation robustness by discovering positive samples with rich intra-class variations from high-density regions in representation space.
• To develop a hard sampling strategy that emphasizes low-similarity positive samples to enhance abstraction and invariance in learned features.
• To achieve state-of-the-art performance on downstream tasks such as linear classification, transfer learning, and semi-supervised learning without human-annotated labels.

Proposed method

• InvP constructs a k-nearest neighbor (k-NN) graph in the embedding space to recursively propagate local invariance and identify semantically consistent positive samples from the same category.
• The method ensures positive samples reside in high-density regions, aligning with the smoothness assumption in semi-supervised learning for improved generalization.
• A hard sampling strategy is introduced to prioritize positive samples with low similarity to the anchor, which provide greater intra-class variation for better invariance learning.
• The model is trained to maximize agreement between the anchor and its hard positive samples while separating negative samples, using a contrastive objective.
• The k-NN graph is updated dynamically during training to reflect evolving representations and maintain semantic consistency.
• The backbone network (e.g., ResNet-50) is optimized end-to-end using a contrastive loss that encourages clustering of positive pairs.

Experimental results

Research questions

• RQ1 Can learning category-level invariance through inter-instance relations improve unsupervised representation learning beyond instance-level invariance?
• RQ2 How does recursive k-NN propagation in representation space help discover semantically consistent positive samples?
• RQ3 Does focusing on hard positive samples—those with low similarity to the anchor—improve the robustness and abstraction of learned representations?
• RQ4 Can InvP achieve state-of-the-art performance on linear evaluation, transfer learning, and semi-supervised learning benchmarks without labeled data?

Key findings

• InvP achieves 71.3% top-1 accuracy on ImageNet linear classification and 78.2% top-5 accuracy when fine-tuning with only 1% labeled data, surpassing previous methods.
• On Places205 and Pascal VOC2007, InvP achieves state-of-the-art results in linear classification, demonstrating strong generalization across datasets.
• The similarity distribution of InvP shows significantly higher and more separable positive similarities (peaking around 0.9) compared to instance discrimination (peaking around 0.5), indicating stronger category-level invariance.
• Qualitative analysis confirms that hard positive samples preserve semantic content while introducing diverse low-level variations (e.g., color, view, pattern), enhancing invariance.
• The method outperforms supervised baselines in some settings, indicating that unsupervised learning with InvP can capture intrinsic data structure effectively.
• Visualization of neighborhoods confirms that InvP successfully groups semantically similar instances from the same category into high-density regions.