[Paper Review] Unsupervised Learning by Predicting Noise

Piotr Bojanowski, Armand Joulin|arXiv (Cornell University)|Apr 18, 2017

Advanced Neural Network ApplicationsComputer Science129 citations

TL;DR

The paper introduces Noise As Targets (NAT), an unsupervised framework that trains convnets end-to-end by mapping features to fixed random target vectors, enabling scalable learning on large datasets like ImageNet. It achieves competitive transfer performance with state-of-the-art unsupervised methods.

ABSTRACT

Convolutional neural networks provide visual features that perform remarkably well in many computer vision applications. However, training these networks requires significant amounts of supervision. This paper introduces a generic framework to train deep networks, end-to-end, with no supervision. We propose to fix a set of target representations, called Noise As Targets (NAT), and to constrain the deep features to align to them. This domain agnostic approach avoids the standard unsupervised learning issues of trivial solutions and collapsing of features. Thanks to a stochastic batch reassignment strategy and a separable square loss function, it scales to millions of images. The proposed approach produces representations that perform on par with state-of-the-art unsupervised methods on ImageNet and Pascal VOC.

Motivation & Objective

• Motivate learning rich visual features without human annotation to avoid bias from labels.
• Propose a scalable, end-to-end discriminative framework that avoids feature collapse.
• Introduce fixed random target representations and a scalable assignment mechanism.
• Demonstrate that NAT yields transferable features comparable to state-of-the-art unsupervised methods.

Proposed method

• Define a mapping f_theta from images to unit-normalized features and jointly optimize theta with fixed target representations.
• Fix k target vectors forming a target matrix C and assign images to targets via a permutation-like matrix P under 1-to-1 constraints.
• Use a square loss between f_theta(X) and Y=PC, where P is updated online to prevent collapsing.
• Choose NAT targets by sampling uniformly on the unit sphere to approximate a distribution over feature space.
• Optimize with SGD and an online Hungarian-based assignment restricted to batch submatrices for scalability.
• Preprocess inputs with image gradients and standard augmentations to improve unsupervised signal quality.

Experimental results

Research questions

• RQ1 Can a discriminative, non-generative unsupervised objective yield transferable visual features?
• RQ2 Does fixing target representations and online reassignment prevent feature collapse while remaining scalable to large datasets?
• RQ3 How do different target representations and update frequencies affect learned representations and transfer performance?
• RQ4 Are NAT-derived features competitive with other unsupervised and self-supervised methods on ImageNet and PASCAL VOC?

Key findings

• NAT achieves transfer performance on ImageNet comparable to state-of-the-art unsupervised and self-supervised approaches.
• Training with square loss on unit-normalized features performs similarly to softmax loss for this setting.
• Discrete, basis-like targets underperform continuous NAT targets in ImageNet transfer.
• Updating target assignments every few epochs provides a good trade-off between performance and computation.
• NAT features visually capture meaningful structures and edges similar to supervised features, though with some differences in sharpness.
• On Pascal VOC 2007, NAT outperforms standard autoencoder and GAN baselines and is competitive with BiGAN, while approaching self-supervised methods.”