[Paper Review] Contrastive Learning with Stronger Augmentations

Xiao Wang, Guo-Jun Qi|arXiv (Cornell University)|Apr 15, 2021

Domain Adaptation and Few-Shot Learning76 references37 citations

TL;DR

Introduces CLSA, a distributional divergence framework that leverages much stronger augmentations by supervising the retrieval of strongly augmented queries via the distribution of weakly augmented views, boosting contrastive-learning performance.

ABSTRACT

Representation learning has significantly been developed with the advance of contrastive learning methods. Most of those methods have benefited from various data augmentations that are carefully designated to maintain their identities so that the images transformed from the same instance can still be retrieved. However, those carefully designed transformations limited us to further explore the novel patterns exposed by other transformations. Meanwhile, as found in our experiments, the strong augmentations distorted the images' structures, resulting in difficult retrieval. Thus, we propose a general framework called Contrastive Learning with Stronger Augmentations~(CLSA) to complement current contrastive learning approaches. Here, the distribution divergence between the weakly and strongly augmented images over the representation bank is adopted to supervise the retrieval of strongly augmented queries from a pool of instances. Experiments on the ImageNet dataset and downstream datasets showed the information from the strongly augmented images can significantly boost the performance. For example, CLSA achieves top-1 accuracy of 76.2% on ImageNet with a standard ResNet-50 architecture with a single-layer classifier fine-tuned, which is almost the same level as 76.5% of supervised results. The code and pre-trained models are available in https://github.com/maple-research-lab/CLSA.

Motivation & Objective

• Motivate exploring stronger, random augmentations beyond carefully designed transformations in contrastive learning.
• Propose a distributional divergence minimization (DDM) loss to supervise retrieval of strongly augmented queries without forcing exact embedding equality.
• Show that combining DDM with existing contrastive losses improves representations for weak and strongly augmented views.
• Demonstrate CLSA’s compatibility with various contrastive frameworks (e.g., MoCo, SimCLR, BYOL) and its impact on downstream tasks.

Proposed method

• Introduce a very strong augmentation S as a random combination of 14 augmentation types.
• Define a weak augmentation pair (T',T) and a strong augmentation S to produce weak and strong views with embeddings z', z, z'' respectively.
• Compute a conditional distribution p(z_j|z') over memory-bank negatives for both weak and strong views (Eqs. 3 and 5).
• Formulate a distributional divergence loss L_D that minimizes the divergence between p(.|z') and p(.|z'') (Eq. 6).
• Combine the standard contrastive loss L_C with L_D: L = L_C + beta * L_D (beta = 1 in experiments) to train the encoder.
• Demonstrate that L_D can be integrated with any concurrent contrastive loss-based method (independent of the base loss).
• Provide implementation details for strong augmentations and ablation studies to support the necessity of distributional supervision.

Figure 1: Contrastive instance learning framework

Experimental results

Research questions

• RQ1 Can stronger, randomly combined augmentations improve self-supervised representations without collapsing performance?
• RQ2 Does a distributional supervision signal between weak and strong views better utilize strong augmentations than naive one-to-one matching?
• RQ3 How does CLSA interact with existing contrastive losses and affect downstream tasks beyond ImageNet linear evaluation?
• RQ4 What are the ablation effects of the distributional loss and the choice of augmentation types on performance?

Key findings

• CLSA achieves strong results on ImageNet with ResNet-50, including a top-1 of 76.2% in linear evaluation, close to supervised 76.5%.
• On VOC07, CLSA pre-training yields 93.6% top-1 with a linear classifier, surpassing the previous record of 88.9%.
• For COCO object detection, CLSA improves AP_S for small objects to 24.4% from 20.8%.
• An ablation study shows naive use of stronger augmentations degrades performance, validating the need for distributional guidance.
• CLSA can significantly boost performance for existing contrastive methods (e.g., MoCo, SimCLR, BYOL) by incorporating the distributional loss.
• The approach also improves representations for strongly augmented images, not just weakly augmented ones.

Figure 2: Comparison of the strongly weakly augmented images. The left is the original image, the middle is the weakly augmented image, and the right is the strongly augmented one with over-contrastive details.