[Paper Review] Object Detectors Emerge in Deep Scene CNNs

Bolei Zhou, Aditya Khosla+3 more|arXiv (Cornell University)|Dec 22, 2014

Advanced Neural Network Applications23 references711 citations

TL;DR

This paper demonstrates that object detectors spontaneously emerge in the internal layers of convolutional neural networks (CNNs) trained solely for scene classification using the Places dataset, without any object-level supervision. The key contribution is that a single CNN can perform both scene recognition and object localization in one forward pass, revealing that object detectors are naturally learned as part of scene representation, with high accuracy in detecting discriminative objects.

ABSTRACT

With the success of new computational architectures for visual processing, such as convolutional neural networks (CNN) and access to image databases with millions of labeled examples (e.g., ImageNet, Places), the state of the art in computer vision is advancing rapidly. One important factor for continued progress is to understand the representations that are learned by the inner layers of these deep architectures. Here we show that object detectors emerge from training CNNs to perform scene classification. As scenes are composed of objects, the CNN for scene classification automatically discovers meaningful objects detectors, representative of the learned scene categories. With object detectors emerging as a result of learning to recognize scenes, our work demonstrates that the same network can perform both scene recognition and object localization in a single forward-pass, without ever having been explicitly taught the notion of objects.

Motivation & Objective

• To investigate whether object detectors can emerge in deep CNNs trained only for scene classification, without explicit object annotations.
• To understand the nature of internal representations in CNNs trained on scene datasets versus object-centric datasets.
• To evaluate whether the internal units of a scene-trained CNN can perform reliable object localization and detection.
• To determine whether the detected objects correspond to those most discriminative for scene categories.
• To demonstrate that a single deep network can support multi-level recognition (edges, textures, objects, scenes) without multiple outputs.

Proposed method

• Training a deep CNN (Places-CNN) on the Places dataset with 205 scene categories, using the same architecture as ImageNet-CNN.
• Analyzing feature maps from the pool5 layer to identify units that activate on specific object classes.
• Using human-annotated tags from Amazon Mechanical Turk to interpret the semantic meaning of each unit's activation.
• Evaluating object detection performance via bounding box localization and segmentation using unit activations.
• Measuring the correlation between object frequency in the dataset and detection frequency in the network.
• Assessing discriminative power by measuring how often each object class is the most informative for scene classification.

Experimental results

Research questions

• RQ1 Can object detectors emerge in a CNN trained only for scene classification, without any object-level supervision?
• RQ2 Do the detected objects in the internal layers correspond to those most discriminative for scene category distinction?
• RQ3 How does the frequency of object detection in the network relate to the frequency of objects in the training dataset?
• RQ4 Can the same network perform both scene recognition and object localization in a single forward pass?
• RQ5 To what extent do internal units in a scene-trained CNN represent meaningful, interpretable object detectors?

Key findings

• Object detectors emerge in the pool5 layer of a CNN trained for scene classification, with 115 out of 262,144 units detecting specific object classes.
• The correlation between object frequency in the SUN dataset and detection frequency in the network is 0.54, indicating a moderate but significant relationship.
• The correlation between object discriminability for scene classification and detection frequency is 0.84, showing that the network prioritizes the most informative objects.
• Segmentation performance for detected objects using pool5 units achieves high Jaccard index and average precision, with many objects showing AP > 0.5.
• The network can perform both scene recognition and object localization in a single forward pass, using only the internal feature maps of the trained network.
• Objects detected are not limited to semantic parts (e.g., eyes, nose) but include scene-relevant objects such as beds, lamps, and cabinets, which are key for scene discrimination.