[Paper Review] RelationNet++: Bridging Visual Representations for Object Detection via Transformer Decoder

Cheng Chi, Fangyun Wei+1 more|arXiv (Cornell University)|Oct 29, 2020

Advanced Neural Network Applications40 references36 citations

TL;DR

RelationNet++ introduces Bridging Visual Representations (BVR), an attention-based module that fuses heterogeneous object representations (e.g., centers, corners, anchors) into a single detector, yielding consistent AP gains across RetinaNet, Faster R-CNN, FCOS, and ATSS, and reaching 52.7 AP on COCO test-dev with a strong backbone.

ABSTRACT

Existing object detection frameworks are usually built on a single format of object/part representation, i.e., anchor/proposal rectangle boxes in RetinaNet and Faster R-CNN, center points in FCOS and RepPoints, and corner points in CornerNet. While these different representations usually drive the frameworks to perform well in different aspects, e.g., better classification or finer localization, it is in general difficult to combine these representations in a single framework to make good use of each strength, due to the heterogeneous or non-grid feature extraction by different representations. This paper presents an attention-based decoder module similar as that in Transformer~\cite{vaswani2017attention} to bridge other representations into a typical object detector built on a single representation format, in an end-to-end fashion. The other representations act as a set of \emph{key} instances to strengthen the main \emph{query} representation features in the vanilla detectors. Novel techniques are proposed towards efficient computation of the decoder module, including a \emph{key sampling} approach and a \emph{shared location embedding} approach. The proposed module is named \emph{bridging visual representations} (BVR). It can perform in-place and we demonstrate its broad effectiveness in bridging other representations into prevalent object detection frameworks, including RetinaNet, Faster R-CNN, FCOS and ATSS, where about $1.5\sim3.0$ AP improvements are achieved. In particular, we improve a state-of-the-art framework with a strong backbone by about $2.0$ AP, reaching $52.7$ AP on COCO test-dev. The resulting network is named RelationNet++. The code will be available at https://github.com/microsoft/RelationNet2.

Motivation & Objective

• Motivate combining multiple object/part representations within a single detector to leverage their complementary strengths.
• Develop a general, end-to-end bridge (BVR) that augments a master representation with auxiliary representations.
• Ensure efficiency with key sampling and shared location embedding to keep inference practical.
• Demonstrate BVR’s plug-in applicability to diverse detectors (RetinaNet, Faster R-CNN, FCOS, ATSS).
• Show gains across datasets (COCO) and state-of-the-art backbones.

Proposed method

• Introduce Bridging Visual Representations (BVR), an attention-based module where the master representation acts as the query and auxiliary representations as keys to strengthen query features.
• Use a multi-head attention formulation with appearance similarity and a geometry term combining relative location embeddings and a small MLP.
• Implement key sampling (top-k) to select a manageable, high-quality set of auxiliary representations.
• Apply shared relative location embedding to compute geometry terms efficiently, reducing time/memory costs.
• Use separate BVR branches for classification (center auxiliary) and regression (corner auxiliary) to enhance each task.
• Integrate BVR in-place with detectors, replacing input features for classification/regression with the enhanced features.

Experimental results

Research questions

• RQ1 Can heterogeneous object/part representations (anchors, centers, corners) be effectively bridged to improve detection without altering the main inference flow?
• RQ2 What are the efficient mechanisms to compute cross-representation interactions (key sampling, shared location embeddings) while controlling compute?
• RQ3 Do separate bridging branches for classification and regression yield better localization and recognition?
• RQ4 How broadly can BVR improve diverse detectors (one-stage, two-stage, anchor-based, anchor-free) on COCO?

Key findings

• BVR improves RetinaNet, Faster R-CNN, FCOS, and ATSS by about 1.5–3.0 AP.
• Using shared keys across pyramid levels yields better gains than independent keys, with 50 keys giving strong results.
• Sub-pixel center/corner representations significantly boost AP, especially for localization.
• Shared relative location embedding substantially reduces memory and FLOPs (e.g., 42× memory savings) while maintaining or improving accuracy.
• Separate BVR modules for classification (center) and regression (corner) outperform shared setups.
• Applying BVR to a strong ATSS backbone reaches 52.7 AP on COCO test-dev (with multi-scale testing).