[Paper Review] Action-Driven Object Detection with Top-Down Visual Attentions

Donggeun Yoo, Sunggyun Park+3 more|arXiv (Cornell University)|Dec 20, 2016

Advanced Neural Network Applications43 references5 citations

TL;DR

This paper proposes an action-driven, top-down object detection method using a visual attention model called AttentionNet that iteratively refines object bounding boxes through sequential actions (e.g., 'move left', 'move down') rather than relying on bottom-up proposals. The method achieves state-of-the-art performance on PASCAL VOC 2007 and ILSVRC CLS-LOC, outperforming Faster R-CNN by +7.1% mAP at IoU=0.7, demonstrating superior localization accuracy through context-aware, action-based refinement.

ABSTRACT

A dominant paradigm for deep learning based object detection relies on a "bottom-up" approach using "passive" scoring of class agnostic proposals. These approaches are efficient but lack of holistic analysis of scene-level context. In this paper, we present an "action-driven" detection mechanism using our "top-down" visual attention model. We localize an object by taking sequential actions that the attention model provides. The attention model conditioned with an image region provides required actions to get closer toward a target object. An action at each time step is weak itself but an ensemble of the sequential actions makes a bounding-box accurately converge to a target object boundary. This attention model we call AttentionNet is composed of a convolutional neural network. During our whole detection procedure, we only utilize the actions from a single AttentionNet without any modules for object proposals nor post bounding-box regression. We evaluate our top-down detection mechanism over the PASCAL VOC series and ILSVRC CLS-LOC dataset, and achieve state-of-the-art performances compared to the major bottom-up detection methods. In particular, our detection mechanism shows a strong advantage in elaborate localization by outperforming Faster R-CNN with a margin of +7.1% over PASCAL VOC 2007 when we increase the IoU threshold for positive detection to 0.7.

Motivation & Objective

• To overcome the limitations of bottom-up object detection methods that rely on class-agnostic proposals and lack holistic scene context.
• To develop a top-down detection mechanism that actively explores object locations using high-level reasoning and visual context.
• To achieve accurate object localization without using region proposal networks or bounding-box regression modules.
• To evaluate the method’s performance under strict IoU thresholds, where localization accuracy is most critical.
• To demonstrate that a single, unified attention model can perform detection, localization, and refinement end-to-end.

Proposed method

• The method uses a top-down search strategy where an object is localized by sequentially applying actions (e.g., move up/down/left/right) to a bounding box based on visual attention from a CNN-based model.
• The AttentionNet model takes an image region as input and outputs a pair of actions for the top-left and bottom-right corners of the current bounding box to converge toward the target object.
• Each action is weak individually but collectively forms a sequence that refines the bounding box through iterative cropping and attention-guided navigation.
• The entire detection pipeline is driven by a single AttentionNet model, eliminating the need for region proposal networks or post-regression modules.
• The method performs end-to-end detection using only the attention model’s action outputs, with no additional detection heads or regression heads.
• A refinement step is introduced as an optional post-processing stage to further improve localization accuracy, mimicking traditional bounding-box regression.

Experimental results

Research questions

• RQ1 Can a top-down, action-driven approach outperform state-of-the-art bottom-up object detection methods in terms of localization accuracy, especially under strict IoU thresholds?
• RQ2 To what extent can a single visual attention model replace multiple components (e.g., region proposal, classifier, regressor) in a standard detection pipeline?
• RQ3 How does the integration of scene-level context through top-down exploration improve detection robustness compared to bottom-up methods?
• RQ4 Does the ensemble of weak actions over time lead to stronger localization than individual actions or traditional regression-based refinement?
• RQ5 Can a single, unified attention model achieve competitive performance on large-scale benchmarks like ILSVRC CLS-LOC without model ensembling?

Key findings

• On PASCAL VOC 2007 with IoU threshold of 0.7, the proposed AttentionNet method achieves 52.3% mAP, outperforming Faster R-CNN by +7.1% mAP.
• The performance drop from IoU=0.5 to IoU=0.7 is only -17.6% for AttentionNet with refinement, significantly lower than the -23.9% drop for Faster R-CNN, indicating superior localization robustness.
• On the ILSVRC CLS-LOC dataset, AttentionNet achieves a Top-5 localization error of 0.1473, outperforming all prior bottom-up methods except the 2015 winner (Faster R-CNN with ResNet-152).
• Even though the 2015 winner has a lower error (0.0902), this is largely due to its superior classification performance (Top-5 CLS error: 0.0357), while AttentionNet's classification error is 0.0792, showing strong localization capability despite simpler architecture.
• The method achieves state-of-the-art results using only a single AttentionNet model without any model ensembling, region proposals, or bounding-box regression.
• The ablation study confirms that the iterative action sequence is key to accurate localization, as the performance drop under high IoU thresholds is significantly lower than in baseline methods.