[Paper Review] Building high-level features using large scale unsupervised learning

Quoc V. Le, Marc’Aurelio Ranzato+6 more|arXiv (Cornell University)|Dec 29, 2011

Face recognition and analysis39 references179 citations

TL;DR

This paper proposes training deep unsupervised autoencoders on 10 million unlabeled YouTube images using a 1,000-machine cluster to learn high-level, class-specific features such as face and body detectors without any labeled data. The model achieves 15.8% top-1 accuracy on ImageNet’s 22,000-category object recognition task, a 70% relative improvement over prior state-of-the-art, demonstrating that complex invariances and high-level concepts can emerge from unsupervised pretraining alone.

ABSTRACT

We consider the problem of building high-level, class-specific feature detectors from only unlabeled data. For example, is it possible to learn a face detector using only unlabeled images? To answer this, we train a 9-layered locally connected sparse autoencoder with pooling and local contrast normalization on a large dataset of images (the model has 1 billion connections, the dataset has 10 million 200x200 pixel images downloaded from the Internet). We train this network using model parallelism and asynchronous SGD on a cluster with 1,000 machines (16,000 cores) for three days. Contrary to what appears to be a widely-held intuition, our experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not. Control experiments show that this feature detector is robust not only to translation but also to scaling and out-of-plane rotation. We also find that the same network is sensitive to other high-level concepts such as cat faces and human bodies. Starting with these learned features, we trained our network to obtain 15.8% accuracy in recognizing 20,000 object categories from ImageNet, a leap of 70% relative improvement over the previous state-of-the-art.

Motivation & Objective

• To investigate whether high-level, class-specific feature detectors can be learned from unlabeled data alone, challenging the assumption that supervision is necessary for such specificity.
• To test the feasibility of training deep neural networks to detect complex visual concepts like faces and human bodies without any labeled examples.
• To scale up unsupervised feature learning by increasing dataset size, model depth, and computational resources to enable discovery of invariances beyond low-level features.
• To evaluate whether unsupervised pretraining with large-scale unlabeled data can significantly improve performance on downstream discriminative tasks like ImageNet object recognition.

Proposed method

• Trained a 9-layer locally connected sparse autoencoder with pooling and local contrast normalization on 10 million 200x200 pixel images from YouTube videos.
• Employed model parallelism and asynchronous stochastic gradient descent (SGD) across a cluster of 1,000 machines (16,000 cores) to scale training to 1 billion parameters.
• Used local receptive fields to reduce inter-machine communication and enable efficient model parallelism in large-scale distributed training.
• Applied unsupervised pretraining to learn hierarchical features before fine-tuning with supervised labels on ImageNet.
• Used numerical optimization and visualization to interpret learned features and validate their selectivity and invariance properties.
• Evaluated performance using standard benchmarks: face detection, human body detection, and ImageNet classification with 22,000 categories.

Experimental results

Research questions

• RQ1 Can high-level, class-specific feature detectors such as face detectors be learned from only unlabeled images, without any bounding box or category annotations?
• RQ2 To what extent do unsupervised features learned from natural video data exhibit invariance to translation, scaling, and out-of-plane rotation?
• RQ3 Can the same unsupervised representation be sensitive to multiple high-level concepts like faces, cat faces, and human bodies?
• RQ4 Does unsupervised pretraining with large-scale unlabeled data lead to significant improvements in downstream discriminative tasks like ImageNet object recognition?
• RQ5 How does the performance of unsupervised features compare to hand-engineered features and random baselines on real-world recognition benchmarks?

Key findings

• The model learned a feature detector highly selective for faces using only 10 million unlabeled YouTube frames, with no labeled data required.
• The learned face detector demonstrated robustness to translation, scaling, and out-of-plane rotation, indicating it captured complex invariances.
• The same network also learned detectors for cat faces and human bodies, showing generalization to multiple high-level visual concepts.
• On the ImageNet 22,000-category benchmark, the method achieved 15.8% top-1 accuracy, a 70% relative improvement over the previous state-of-the-art.
• The unsupervised pretraining approach outperformed random initialization and linear filter baselines, achieving 74.8% and 76.7% accuracy on cat and human body detection tasks, respectively.
• The results confirm that high-level, class-specific neurons—inspired by the 'grandmother neuron' hypothesis—can emerge from unsupervised learning on large-scale unlabeled data.