[論文レビュー] Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Senthil Purushwalkam, Abhinav Gupta|arXiv (Cornell University)|Jul 28, 2020

Domain Adaptation and Few-Shot Learning参考文献 40被引用数 130

ひとこと要約

本論文は対照学習型自己監視学習がなぜ成功するのかを分析し、遮蔽不変性の大部分が過度なデータ拡張とオブジェクト中心のデータセットバイアスに起因することを示し、視点不変性を改善するための動画ベースの時系列変換を提案する。

ABSTRACT

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

研究の動機と目的

• Investigate what invariances are encoded by contrastive self-supervised representations in object recognition tasks.
• Analyze the role of data augmentation strategies and dataset biases in the success of contrastive SSL methods.
• Evaluate how self-supervised methods compare to supervised baselines across key invariances (occlusion, viewpoint, illumination, instance).
• Propose and test alternatives (using videos) to improve viewpoint and other invariances in learned representations.

提案手法

• Formalize contrastive learning objective and positive/negative pair construction.
• Quantify invariances via a Top-K Representation Invariance Score (RIS) across occlusion, viewpoint, illumination, and instance factors.
• Diagnose the impact of augmentation schemes (random crops, aggressive Cropping) and dataset biases (ImageNet object-centric bias) on learned representations.
• Compare supervised vs. self-supervised (MOCOv2, PIRL) representations on downstream tasks and invariances.
• Propose temporal transformation-based learning from videos (frame-level and region-tracking) to enhance viewpoint and illumination invariances.
• Evaluate proposed video-based representations on classification (Pascal, Pascal Cropped Boxes, ImageNet) and segmentation (ADE20K).

実験結果

リサーチクエスチョン

• RQ1 What invariances do contrastive self-supervised representations encode and how do these relate to the augmentations used during pre-training?
• RQ2 To what extent do self-supervised methods achieve occlusion, viewpoint, and instance invariance compared to supervised baselines?
• RQ3 How do data biases in pre-training datasets (e.g., ImageNet's object-centric bias) affect learned representations and downstream performance?
• RQ4 Can leveraging temporally coherent transformations from videos improve viewpoint, deformation, and other invariances in representations?
• RQ5 Do video-based or region-tracking approaches yield representations that outperform image-based MOCOv2 on invariance and downstream tasks?

主な発見

Dataset	Method	Occlusion Top-10	Occlusion Top-25	Viewpoint Top-10	Viewpoint Top-25	Illumination Dir. Top-10	Illumination Dir. Top-25	Illumination Color Top-10	Illumination Color Top-25	Instance Top-10	Instance Top-25	Instance+Viewpoint Top-10	Instance+Viewpoint Top-25
Imagenet	Sup. R50	80.89	74.21	89.54	82.62	94.63	89.08	99.88	99.38	66.11	59.44	70.17	63.47
Imagenet	MOCOv2	84.19	77.88	85.15	75.08	90.28	80.76	99.66	97.11	62.49	55.01	67.40	60.52
Imagenet	PIRL	84.46	78.38	85.8	76.08	85.?	?	99.68	97.19	52.97	46.79	57.01	51.03

• Self-supervised methods (MOCO, PIRL) exhibit strong occlusion invariance due to aggressive cropping, but lag behind supervised models in viewpoint and instance invariances.
• Occlusion invariance from aggressive augmentation is not necessarily beneficial for all tasks, and reliance on object-centric dataset biases may drive observed gains.
• Supervised models on ImageNet show different invariance profiles, with self-supervised methods excelling in occlusion while underperforming in viewpoint, illumination direction/color, and instance invariance.
• Evaluations on MSCOCO and cropped-box variants reveal that object-centric biases in pre-training data significantly influence discriminative power and transferability.
• Video-based temporal transformations (frame-level and region-tracker approaches) improve viewpoint and illumination invariances and can surpass MOCOv2 trained on the same data in several metrics.
• Region-tracker representations achieve higher viewpoint and illumination invariance and competitive downstream performance (Pascal, ImageNet, ADE20K) compared to frame-based methods.