[Paper Review] Unsupervised Representation Learning by Predicting Image Rotations

Spyros Gidaris, Praveer Singh+1 more|arXiv (Cornell University)|Mar 21, 2018

Advanced Image and Video Retrieval Techniques25 references1,534 citations

TL;DR

The paper learns semantic image features by training a ConvNet to predict 0/90/180/270 degree rotations, achieving state-of-the-art unsupervised transfer and semi-supervised results across CIFAR-10, ImageNet, PASCAL, and Places.

ABSTRACT

Over the last years, deep convolutional neural networks (ConvNets) have transformed the field of computer vision thanks to their unparalleled capacity to learn high level semantic image features. However, in order to successfully learn those features, they usually require massive amounts of manually labeled data, which is both expensive and impractical to scale. Therefore, unsupervised semantic feature learning, i.e., learning without requiring manual annotation effort, is of crucial importance in order to successfully harvest the vast amount of visual data that are available today. In our work we propose to learn image features by training ConvNets to recognize the 2d rotation that is applied to the image that it gets as input. We demonstrate both qualitatively and quantitatively that this apparently simple task actually provides a very powerful supervisory signal for semantic feature learning. We exhaustively evaluate our method in various unsupervised feature learning benchmarks and we exhibit in all of them state-of-the-art performance. Specifically, our results on those benchmarks demonstrate dramatic improvements w.r.t. prior state-of-the-art approaches in unsupervised representation learning and thus significantly close the gap with supervised feature learning. For instance, in PASCAL VOC 2007 detection task our unsupervised pre-trained AlexNet model achieves the state-of-the-art (among unsupervised methods) mAP of 54.4% that is only 2.4 points lower from the supervised case. We get similarly striking results when we transfer our unsupervised learned features on various other tasks, such as ImageNet classification, PASCAL classification, PASCAL segmentation, and CIFAR-10 classification. The code and models of our paper will be published on: https://github.com/gidariss/FeatureLearningRotNet .

Motivation & Objective

• Motivate unsupervised semantic feature learning without manual labeling.
• Propose a self-supervised task: predict image rotation among 0/90/180/270 degrees.
• Demonstrate that rotation-based supervision yields transferable features across datasets and tasks.
• Evaluate under CIFAR-10, ImageNet, PASCAL VOC, and Places205 in supervised, semi-supervised, and transfer settings.
• Show that learned features approach supervised performance on several tasks.

Proposed method

• Define a set of K discrete geometric transformations G as image rotations by multiples of 90 degrees (0, 90, 180, 270).
• Train a ConvNet F(.) to predict the rotation label y given a rotated image X^y, forming a 4-way classification task.
• Optimize the loss over N images: loss(X_i,θ) = - (1/K) sum_y log(F^y(g(X_i|y)|θ)).
• Implement rotations using flip and transpose operations to avoid low-level artifacts.
• Visualize attention maps and first-layer filters to argue that rotation prediction requires semantic understanding.
• Evaluate RotNet features by transferring them to CIFAR-10, ImageNet, PASCAL VOC, and Places205 tasks.
• Compare RotNet with prior unsupervised methods and with supervised baselines, including semi-supervised setups.

Experimental results

Research questions

• RQ1 Can a simple self-supervised task of predicting image rotations learn semantically meaningful features?
• RQ2 How do rotation-based features transfer to image classification, detection, and segmentation tasks across diverse datasets?
• RQ3 What is the impact of model depth and the number of rotation classes on feature quality?
• RQ4 How do rotation-based features perform in semi-supervised learning settings compared to fully supervised baselines?
• RQ5 Do the learned features require extensive pre-processing or special artifacts to be avoided?

Key findings

• RotNet achieves state-of-the-art results among unsupervised methods across CIFAR-10, ImageNet, PASCAL VOC, and Places205.
• On CIFAR-10, RotNet features with a 4-block network yield up to 89.06% accuracy when used with a non-linear classifier, close to supervised performance (92.80%).
• RotNet features provide strong transfer in ImageNet top-1 classification with non-linear and linear probes, surpassing prior unsupervised approaches by notable margins.
• In semi-supervised CIFAR-10 experiments, RotNet-based features outperform supervised counterparts when labeled data per class is scarce (below ~1000 examples).
• RotNet features learned on ImageNet transfer effectively to PASCAL VOC classification/detection and Places classification, with substantial gains over prior unsupervised methods.