[Paper Review] Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases

Senthil Purushwalkam, Abhinav Gupta|arXiv (Cornell University)|Jul 28, 2020

Domain Adaptation and Few-Shot LearningComputer Science40 references130 citations

TL;DR

The paper analyzes why contrastive self-supervised learning succeeds, showing occlusion invariance largely comes from aggressive augmentations and object-centric dataset biases, and proposes video-based temporal transformations to improve viewpoint invariance.

ABSTRACT

Self-supervised representation learning approaches have recently surpassed their supervised learning counterparts on downstream tasks like object detection and image classification. Somewhat mysteriously the recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. In this work, we first present quantitative experiments to demystify these gains. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. However, they fail to capture viewpoint and category instance invariance which are crucial components for object recognition. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet. Finally, we propose an approach to leverage unstructured videos to learn representations that possess higher viewpoint invariance. Our results show that the learned representations outperform MOCOv2 trained on the same data in terms of invariances encoded and the performance on downstream image classification and semantic segmentation tasks.

Motivation & Objective

• Investigate what invariances are encoded by contrastive self-supervised representations in object recognition tasks.
• Analyze the role of data augmentation strategies and dataset biases in the success of contrastive SSL methods.
• Evaluate how self-supervised methods compare to supervised baselines across key invariances (occlusion, viewpoint, illumination, instance).
• Propose and test alternatives (using videos) to improve viewpoint and other invariances in learned representations.

Proposed method

• Formalize contrastive learning objective and positive/negative pair construction.
• Quantify invariances via a Top-K Representation Invariance Score (RIS) across occlusion, viewpoint, illumination, and instance factors.
• Diagnose the impact of augmentation schemes (random crops, aggressive Cropping) and dataset biases (ImageNet object-centric bias) on learned representations.
• Compare supervised vs. self-supervised (MOCOv2, PIRL) representations on downstream tasks and invariances.
• Propose temporal transformation-based learning from videos (frame-level and region-tracking) to enhance viewpoint and illumination invariances.
• Evaluate proposed video-based representations on classification (Pascal, Pascal Cropped Boxes, ImageNet) and segmentation (ADE20K).

Experimental results

Research questions

• RQ1 What invariances do contrastive self-supervised representations encode and how do these relate to the augmentations used during pre-training?
• RQ2 To what extent do self-supervised methods achieve occlusion, viewpoint, and instance invariance compared to supervised baselines?
• RQ3 How do data biases in pre-training datasets (e.g., ImageNet's object-centric bias) affect learned representations and downstream performance?
• RQ4 Can leveraging temporally coherent transformations from videos improve viewpoint, deformation, and other invariances in representations?
• RQ5 Do video-based or region-tracking approaches yield representations that outperform image-based MOCOv2 on invariance and downstream tasks?

Key findings

• Self-supervised methods (MOCO, PIRL) exhibit strong occlusion invariance due to aggressive cropping, but lag behind supervised models in viewpoint and instance invariances.
• Occlusion invariance from aggressive augmentation is not necessarily beneficial for all tasks, and reliance on object-centric dataset biases may drive observed gains.
• Supervised models on ImageNet show different invariance profiles, with self-supervised methods excelling in occlusion while underperforming in viewpoint, illumination direction/color, and instance invariance.
• Evaluations on MSCOCO and cropped-box variants reveal that object-centric biases in pre-training data significantly influence discriminative power and transferability.
• Video-based temporal transformations (frame-level and region-tracker approaches) improve viewpoint and illumination invariances and can surpass MOCOv2 trained on the same data in several metrics.
• Region-tracker representations achieve higher viewpoint and illumination invariance and competitive downstream performance (Pascal, ImageNet, ADE20K) compared to frame-based methods.