[Paper Review] Adversarial Masking for Self-Supervised Learning

Yuge Shi, N. Siddharth+2 more|arXiv (Cornell University)|Jan 31, 2022

Domain Adaptation and Few-Shot Learning14 citations

TL;DR

ADIOS proposes a novel self-supervised learning framework that jointly trains a masking function and image encoder via an adversarial objective, where the encoder minimizes representation distance between original and masked images, while the masking function maximizes it. The method generates semantically meaningful masks, consistently improving performance over state-of-the-art SSL methods across multiple benchmarks including ImageNet100, CIFAR, and iNaturalist, and works with convolutional backbones without requiring Vision Transformers or image decoders.

ABSTRACT

We propose ADIOS, a masked image model (MIM) framework for self-supervised learning, which simultaneously learns a masking function and an image encoder using an adversarial objective. The image encoder is trained to minimise the distance between representations of the original and that of a masked image. The masking function, conversely, aims at maximising this distance. ADIOS consistently improves on state-of-the-art self-supervised learning (SSL) methods on a variety of tasks and datasets -- including classification on ImageNet100 and STL10, transfer learning on CIFAR10/100, Flowers102 and iNaturalist, as well as robustness evaluated on the backgrounds challenge (Xiao et al., 2021) -- while generating semantically meaningful masks. Unlike modern MIM models such as MAE, BEiT and iBOT, ADIOS does not rely on the image-patch tokenisation construction of Vision Transformers, and can be implemented with convolutional backbones. We further demonstrate that the masks learned by ADIOS are more effective in improving representation learning of SSL methods than masking schemes used in popular MIM models. Code is available at https://github.com/YugeTen/adios.

Motivation & Objective

• To address the limitation of random or patchwise masking in masked image modeling (MIM), which relies on local pixel correlations rather than semantic reasoning.
• To improve self-supervised representation learning by learning masks that occlude whole semantic entities, akin to word masking in NLP.
• To develop a framework that is compatible with both ViT and convolutional backbones, avoiding dependency on patch-based tokenization.
• To demonstrate that semantically meaningful masks significantly outperform random or fixed-shaped masks in improving SSL performance.
• To show that the adversarial training of the masking function leads to more effective augmentation for representation learning.

Proposed method

• ADIOS employs a Siamese-style adversarial framework with two components: an inference model (encoder) and an occlusion model (mask generator).
• The occlusion model generates image-sized masks with values in [0,1], where 1 indicates masked regions, using a U-Net-based architecture.
• The inference model is trained to minimize the contrastive loss between the representations of the original image and the masked image.
• The occlusion model is trained adversarially to maximize the same contrastive loss, making it harder for the encoder to distinguish between original and masked inputs.
• Masks are refined using fully connected conditional random fields (CRF) to improve spatial coherence and semantic consistency.
• The framework is implemented as an encoder-only model, avoiding the need for a decoder or visual tokenizers, enabling compatibility with convolutional backbones.

Experimental results

Research questions

• RQ1 Can adversarially trained masking functions that generate semantically meaningful masks improve self-supervised representation learning compared to random or fixed-shaped masking?
• RQ2 Does the performance gain from semantic masking hold across diverse datasets and downstream tasks, including ImageNet100, CIFAR, and iNaturalist?
• RQ3 Can ADIOS be effectively applied with convolutional neural network backbones, avoiding reliance on Vision Transformers?
• RQ4 How does the quality of the learned masks compare to ground-truth object masks in terms of representation learning performance?
• RQ5 Is the adversarial masking objective functionally superior to generative reconstruction objectives in self-supervised learning?

Key findings

• ADIOS achieves state-of-the-art performance on ImageNet100 classification, outperforming existing MIM methods including MAE and BEiT.
• On CIFAR-10 and CIFAR-100, ADIOS improves linear probe accuracy by up to 2.5% compared to baseline MIM methods.
• Transfer learning on iNaturalist and Flowers102 shows consistent gains, with improvements of over 3% on iNaturalist-2021.
• Masks generated by ADIOS are semantically meaningful and nearly as effective as ground-truth object masks in boosting representation learning.
• The ablation study confirms that semantic masks significantly outperform random or fixed-shaped masks, validating the importance of mask design in MIM.
• ADIOS improves performance across multiple SSL objectives, including contrastive and non-contrastive methods, demonstrating its generalizability.