[Paper Review] Analyzing the Performance of Multilayer Neural Networks for Object Recognition

Pulkit Agrawal, Ross Girshick+1 more|arXiv (Cornell University)|Jul 7, 2014

Advanced Neural Network Applications22 references71 citations

TL;DR

This paper empirically analyzes multilayer CNNs for object recognition, revealing that pre-training on ImageNet significantly boosts performance even with moderate detection data, features are largely distributed rather than grandmother-cell-like, and spatial location matters more for detection than classification. Surprisingly, feature magnitude is largely unimportant, as binarizing features degrades performance minimally, enabling efficient sparse representations.

ABSTRACT

In the last two years, convolutional neural networks (CNNs) have achieved an impressive suite of results on standard recognition datasets and tasks. CNN-based features seem poised to quickly replace engineered representations, such as SIFT and HOG. However, compared to SIFT and HOG, we understand much less about the nature of the features learned by large CNNs. In this paper, we experimentally probe several aspects of CNN feature learning in an attempt to help practitioners gain useful, evidence-backed intuitions about how to apply CNNs to computer vision problems.

Motivation & Objective

• To understand the nature of features learned by deep CNNs, especially in comparison to handcrafted features like SIFT and HOG.
• To investigate the impact of supervised pre-training and fine-tuning on object recognition performance across varying data regimes.
• To examine whether intermediate CNN features resemble 'grandmother cells' or form distributed codes.
• To evaluate the importance of spatial location and activation magnitude in CNN features for classification and detection tasks.

Proposed method

• Fine-tuned a deep CNN (AlexNet-like) on PASCAL VOC 2007 and SUN datasets using transfer learning from ImageNet.
• Trained the network from random initialization to compare with pre-trained models, varying training data size.
• Applied ablation techniques: binarizing feature maps (threshold 0), spatial shuffling (sp-shuffle), and spatial max-pooling (sp-max) to assess feature robustness.
• Used entropy curves and area under the curve (AuE) to measure filter discriminative capacity, sorting filters by class selectivity.
• Performed experiments on both image classification (PASCAL-CLS, SUN-CLS) and object detection (PASCAL-DET) to compare sensitivity to spatial and magnitude ablations.
• Reported performance using mAP for detection and accuracy for classification, with statistical significance assessed via standard errors.

Experimental results

Research questions

• RQ1 Does supervised pre-training on ImageNet provide consistent performance gains across different data regimes, including moderate-sized detection datasets?
• RQ2 Does longer pre-training on ImageNet lead to overfitting or generalization error when transferring to downstream tasks?
• RQ3 Do intermediate CNN features exhibit 'grandmother cell' behavior, or is the representation more distributed across multiple filters?
• RQ4 How critical is the spatial location of feature activations for image classification versus object detection?
• RQ5 To what extent does the magnitude of feature activations affect recognition performance?

Key findings

• Pre-training on ImageNet significantly improves detection and classification performance even with as few as 37,000 bounding boxes, and longer pre-training yields better results without increasing generalization error.
• On PASCAL-CLS, binarizing features (threshold 0) reduced mAP from 65.6% to 60.8% on conv-5, indicating minimal performance loss and supporting the use of sparse binary features.
• Spatial ablation (sp-max) had minimal impact on classification performance—conv-5 mAP dropped only from 65.6% to 62.5%—but caused a large drop in detection (from 47.6% to 25.4%), showing spatial location is critical for detection.
• The representation in intermediate layers is predominantly distributed: only a small number of filters act like 'grandmother cells', while most require coordinated activation across multiple filters for class discrimination.
• For image classification, spatial scrambling (sp-shuffle) reduced performance more in early layers (e.g., conv-1 from 25.1% to 15.1% mAP), but the effect diminished in deeper layers, indicating spatial information becomes less critical downstream.
• On ImageNet validation data, conv-5 features after sp-max achieved 41.5% top-1 accuracy, close to the original 43.2%, confirming that spatial structure is less critical for classification than for detection.