[Paper Review] Beyond Skip Connections: Top-Down Modulation for Object Detection

Abhinav Shrivastava, Rahul Sukthankar+2 more|arXiv (Cornell University)|Dec 20, 2016

Advanced Image and Video Retrieval Techniques43 references336 citations

TL;DR

Introduces a Top-Down Modulation (TDM) network that augments bottom-up ConvNets with a top-down pathway and lateral connections to preserve fine details for improved COCO object detection.

ABSTRACT

In recent years, we have seen tremendous progress in the field of object detection. Most of the recent improvements have been achieved by targeting deeper feedforward networks. However, many hard object categories such as bottle, remote, etc. require representation of fine details and not just coarse, semantic representations. But most of these fine details are lost in the early convolutional layers. What we need is a way to incorporate finer details from lower layers into the detection architecture. Skip connections have been proposed to combine high-level and low-level features, but we argue that selecting the right features from low-level requires top-down contextual information. Inspired by the human visual pathway, in this paper we propose top-down modulations as a way to incorporate fine details into the detection framework. Our approach supplements the standard bottom-up, feedforward ConvNet with a top-down modulation (TDM) network, connected using lateral connections. These connections are responsible for the modulation of lower layer filters, and the top-down network handles the selection and integration of contextual information and low-level features. The proposed TDM architecture provides a significant boost on the COCO testdev benchmark, achieving 28.6 AP for VGG16, 35.2 AP for ResNet101, and 37.3 for InceptionResNetv2 network, without any bells and whistles (e.g., multi-scale, iterative box refinement, etc.).

Motivation & Objective

• Motivate the need to preserve fine-detail features from early CNN layers for accurate object detection.
• Propose a top-down modulation framework to selectively transmit high-level context to lower layers.
• Demonstrate end-to-end trainability of TDM integrated with standard detection pipelines.
• Show that TDM yields consistent gains across multiple backbone architectures on COCO.

Proposed method

• Add a top-down modulation (TDM) network to a bottom-up ConvNet, connected via lateral modules.
• Use lateral modules to transform bottom-up features and top-down modules to fuse and upsample context with low-level features.
• Train the entire TDM-enhanced detector end-to-end within the Faster R-CNN framework.
• Vary capacities of T, L, and T_out modules to control representation size and ensure compatibility with RPN/RCN heads.
• Progressively add TDM pairs (L_i, T_{i+1,i}) starting from higher to lower layers during training.
• Demonstrate that TDM improves detection across VGG16, ResNet101, and InceptionResNetv2 backbones; provide ablations.

Experimental results

Research questions

• RQ1 Does top-down modulation improve object detection performance by preserving fine-grained details from early CNN layers?
• RQ2 How should the top-down and lateral modules be designed (capacity, placement, upsampling) to maximize detector performance?
• RQ3 Can TDM be trained end-to-end with existing detectors like Faster R-CNN across diverse backbones?
• RQ4 What is the impact of TDM on small objects and localization accuracy (AP, AP75) across architectures?

Key findings

• TDM yields substantial AP gains across backbones: VGG16 + TDM 28.6 AP vs 23.3 AP baseline.
• ResNet101 + TDM 35.2 AP vs 31.5 AP baseline.
• InceptionResNetv2 + TDM 37.3 AP vs 34.7 AP baseline.
• On COCO testdev, TDM with IRNv2 achieves 37.3 AP, the best single-model result without bells and whistles at that time.
• TDM improves small-object detection (AP^S) and localization (AP^75) significantly across architectures.
• Ablations show benefits from incorporating top-down context and selective low-level feature modulation.