[Paper Review] Selfie: Self-supervised Pretraining for Image Embedding

Trieu H. Trinh, Minh-Thang Luong+1 more|arXiv (Cornell University)|Jun 7, 2019

Multimodal Machine Learning Applications42 references76 citations

TL;DR

Selfie pretrains image encoders using a masked-patch prediction task with distractor patches from the same image, improving downstream accuracy and training stability, especially with limited labeled data.

ABSTRACT

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-out patches in an input image, our method learns to select the correct patch, among other "distractor" patches sampled from the same image, to fill in the masked location. This classification objective sidesteps the need for predicting exact pixel values of the target patches. The pretraining architecture of Selfie includes a network of convolutional blocks to process patches followed by an attention pooling network to summarize the content of unmasked patches before predicting masked ones. During finetuning, we reuse the convolutional weights found by pretraining. We evaluate Selfie on three benchmarks (CIFAR-10, ImageNet 32 x 32, and ImageNet 224 x 224) with varying amounts of labeled data, from 5% to 100% of the training sets. Our pretraining method provides consistent improvements to ResNet-50 across all settings compared to the standard supervised training of the same network. Notably, on ImageNet 224 x 224 with 60 examples per class (5%), our method improves the mean accuracy of ResNet-50 from 35.6% to 46.7%, an improvement of 11.1 points in absolute accuracy. Our pretraining method also improves ResNet-50 training stability, especially on low data regime, by significantly lowering the standard deviation of test accuracies across different runs.

Motivation & Objective

• Motivate reducing labeled-data requirements for image models by leveraging unlabeled data.
• Extend masked language modeling concepts to continuous image data.
• Propose a patch-based encoder–decoder with contrastive classification to fill masked regions.
• Enable efficient pretraining by reusing partial networks during finetuning.
• Demonstrate gains across CIFAR-10, ImageNet-32, and ImageNet-224 under low-label conditions.

Proposed method

• Encode image patches with a patch processing network P (first three blocks of ResNet-50).
• Pool patch representations with an attention pooling network A (Transformer-based).
• Mask a subset of image patches and task the decoder to identify the correct patch among distractors from the same image using a cross-entropy loss.
• Train encoder and decoder jointly; during fine-tuning reuse pretrained P and apply full-image ResNet-50 with end-to-end fine-tuning.
• Use positional embeddings for patches (image-size dependent) and partial parameter sharing to reduce computation.
• During pretraining, the decoder predicts multiple correct patches simultaneously to reuse encoder computations.

Experimental results

Research questions

• RQ1 Does self-supervised pretraining with patch-level masking and distractors improve image representations for downstream tasks?
• RQ2 How does Selfie perform under varying labeled-data regimes compared to fully supervised baselines?
• RQ3 What is the impact of pretraining on training stability and variability across runs?
• RQ4 What are the effects of using attention pooling and hybrid convolution-attention architectures on finetuning?
• RQ5 How does unlabeled data abundance relative to labeled data affect gains from Selfie?

Key findings

• Selfie yields consistent accuracy gains across CIFAR-10, ImageNet-32, and ImageNet-224 as labeled data decreases.
• On ImageNet-224×224 with 5% labeled data, accuracy improves from 35.6% (Supervised) to 46.7% (Selfie), an 11.1-point gain.
• Pretraining reduces test accuracy variability and improves training stability, especially in low-data regimes.
• ImageNet-224×224 with 60 labeled examples per class shows substantial gains (11.1 points) and smaller gains as labeled data increases.
• A hybrid ResNet-36 + attention pooling finetuned with Selfie can outperform ResNet-50 in some low-data scenarios.
• Selfie outperforms prior unsupervised pretraining results on ImageNet, indicating effective use of unlabeled data for representation learning.