[Paper Review] Contrastive Learning with Adversarial Examples

Chih-Hui Ho, Nuno Vasconcelos|arXiv (Cornell University)|Oct 22, 2020

Domain Adaptation and Few-Shot Learning77 references54 citations

TL;DR

Introduces CLAE, a method that uses adversarial augmentations to create harder positive and negative pairs in contrastive learning for SSL, improving downstream classification across several baselines and datasets.

ABSTRACT

Contrastive learning (CL) is a popular technique for self-supervised learning (SSL) of visual representations. It uses pairs of augmentations of unlabeled training examples to define a classification task for pretext learning of a deep embedding. Despite extensive works in augmentation procedures, prior works do not address the selection of challenging negative pairs, as images within a sampled batch are treated independently. This paper addresses the problem, by introducing a new family of adversarial examples for constrastive learning and using these examples to define a new adversarial training algorithm for SSL, denoted as CLAE. When compared to standard CL, the use of adversarial examples creates more challenging positive pairs and adversarial training produces harder negative pairs by accounting for all images in a batch during the optimization. CLAE is compatible with many CL methods in the literature. Experiments show that it improves the performance of several existing CL baselines on multiple datasets.

Motivation & Objective

• Motivate the need for harder negative/positive pairs in contrastive self-supervised learning (SSL).
• Propose an adversarial augmentation framework to generate challenging pairs for CL.
• Develop CLAE, a training algorithm that combines adversarially perturbed positives with standard negatives in SSL.
• Demonstrate that CLAE improves downstream classification performance across multiple CL baselines and datasets.

Proposed method

• Define adversarial augmentations that maximize the contrastive loss by perturbing augmentations in a batch-aware manner.
• Replace conventional positive augmentations with adversarially perturbed augmentations that are optimized with respect to the cross-entropy loss.
• Train using a two-branch scheme inspired by AdvProp with separate batch normalization for clean and adversarial examples, and balance losses via a hyperparameter alpha.
• Compute adversarial perturbations by FGSM-type updates on the input of the augmentation, propagating through the batch to account for all instances.
• Integrate CLAE with existing CL losses (Plain, UEL, SimCLR) and show compatibility with various CL frameworks.

Experimental results

Research questions

• RQ1 Can adversarial augmentations that jointly optimize positive and negative pairs improve contrastive SSL representations?
• RQ2 Does adversarial training in CL (CLA E) yield better downstream classifiers than standard CL baselines across datasets?
• RQ3 How do batch-aware adversarial perturbations affect hard negative mining in SSL compared to traditional methods?
• RQ4 Is CLAE robust to different attack methods and network architectures while remaining beneficial across datasets?

Key findings

• Adversarial augmentations elicit larger contrastive and downstream cross-entropy losses than random perturbations, making training pairs more challenging.
• CLA E improves downstream classification accuracy for all tested CL baselines on CIFAR-10, CIFAR-100, and tinyImageNet across kNN and LR classifiers.
• Adversarial training yields gains with epsilon values around 0.03–0.07, with improvements evident even at smaller batch sizes and stronger architectures.
• Using AdvProp-style dual batch normalization, CLAE maintains stable performance and benefits are observed with various attack methods (FGSM, R-FGSM, F-FGSM, PGD).
• Longer pretext training and larger architectures amplify CLAE gains more than baseline CL methods, indicating better utilization of challenging training pairs.