[Paper Review] Center-wise Local Image Mixture For Contrastive Representation Learning

Hao Li, Xiaopeng Zhang+1 more|arXiv (Cornell University)|Nov 5, 2020

Domain Adaptation and Few-Shot Learning33 references4 citations

TL;DR

This paper proposes CLIM, a contrastive representation learning method that improves feature generalization by selecting semantically similar positive samples via center-wise local image mixture. By combining cluster-center-guided positive sampling with cutmix-based data mixing and multi-resolution augmentation, CLIM achieves 75.5% top-1 accuracy on ImageNet linear evaluation with ResNet-50, outperforming MoCo by 4.8% with only 200 training epochs.

ABSTRACT

Contrastive learning based on instance discrimination trains model to discriminate different transformations of the anchor sample from other samples, which does not consider the semantic similarity among samples. This paper proposes a new kind of contrastive learning method, named CLIM, which uses positives from other samples in the dataset. This is achieved by searching local similar samples of the anchor, and selecting samples that are closer to the corresponding cluster center, which we denote as center-wise local image selection. The selected samples are instantiated via an data mixture strategy, which performs as a smoothing regularization. As a result, CLIM encourages both local similarity and global aggregation in a robust way, which we find is beneficial for feature representation. Besides, we introduce \emph{multi-resolution} augmentation, which enables the representation to be scale invariant. We reach 75.5% top-1 accuracy with linear evaluation over ResNet-50, and 59.3% top-1 accuracy when fine-tuned with only 1% labels.

Motivation & Objective

• To address the limitation of instance discrimination in contrastive learning, which ignores semantic similarity among samples.
• To improve feature representation by incorporating both local similarity and global clustering structure in self-supervised learning.
• To reduce the impact of noisy positive samples in weakly supervised contrastive learning through robust data mixing regularization.
• To explicitly model scale invariance in representations using multi-resolution data augmentation.
• To achieve state-of-the-art performance with minimal labeled data, particularly under low-shot fine-tuning settings.

Proposed method

• Proposes center-wise local image selection: for each anchor image, identify its k-nearest neighbors and select only those closer to the corresponding cluster center as positive samples.
• Introduces a data mixing strategy (cutmix-style) to regularize the contrastive loss, smoothing predictions and reducing overconfidence on noisy positives.
• Employs multi-resolution augmentation by contrasting the same image patch at different resolutions (e.g., 224×224 and 160×160), explicitly modeling scale invariance.
• Uses K-means clustering to define cluster centers, which guide the selection of semantically relevant positive samples.
• Applies a standard contrastive loss (e.g., MoCo-style) on the mixed representations, combining the benefits of local similarity and global clustering.
• Optimizes the model end-to-end using standard self-supervised training with standard data augmentations (crop, color jitter, etc.) alongside the proposed components.

Experimental results

Research questions

• RQ1 Can selecting positive samples based on proximity to cluster centers improve representation learning beyond instance discrimination?
• RQ2 Does data mixing via cutmix effectively regularize contrastive learning when positive samples contain noise?
• RQ3 Can explicit multi-resolution contrastive learning enhance scale invariance and performance in self-supervised representation learning?
• RQ4 How does CLIM perform under low-data fine-tuning settings, particularly with only 1% labeled data?
• RQ5 Does the combination of center-wise sampling, data mixing, and multi-resolution augmentation lead to consistent gains across benchmarks?

Key findings

• CLIM achieves 75.5% top-1 accuracy on ImageNet linear evaluation using ResNet-50, surpassing MoCo v2 by 4.8% with only 200 training epochs.
• With only 1% labeled data, CLIM achieves 59.3% top-1 accuracy when fine-tuned, outperforming fully supervised baselines on downstream tasks.
• Center-wise positive sampling improves performance by 1.3% over KNN+K-means selection and 1.8% over standard MoCo v2.
• Cutmix-based data mixing consistently improves accuracy across all sampling strategies, increasing performance by up to 1.8% compared to no mixing.
• Multi-resolution augmentation with 224×224 and 160×160 crops yields 72.3% accuracy, significantly outperforming multi-crop baselines (69.7%) and MoCo with 800 epochs (71.1%).
• Ablation studies confirm that the combination of center-wise selection, cutmix, and multi-resolution augmentation yields the strongest performance, with each component contributing incrementally.