[Paper Review] Relation Networks for Object Detection

Han Hu, Jiayuan Gu+3 more|arXiv (Cornell University)|Nov 30, 2017

Advanced Neural Network Applications56 references67 citations

TL;DR

The paper introduces an object relation module that models interactions among detected objects using appearance and geometry, enabling end-to-end object detection improvements in CNN-based pipelines, including instance recognition and duplicate removal.

ABSTRACT

Although it is well believed for years that modeling relations between objects would help object recognition, there has not been evidence that the idea is working in the deep learning era. All state-of-the-art object detection systems still rely on recognizing object instances individually, without exploiting their relations during learning. This work proposes an object relation module. It processes a set of objects simultaneously through interaction between their appearance feature and geometry, thus allowing modeling of their relations. It is lightweight and in-place. It does not require additional supervision and is easy to embed in existing networks. It is shown effective on improving object recognition and duplicate removal steps in the modern object detection pipeline. It verifies the efficacy of modeling object relations in CNN based detection. It gives rise to the first fully end-to-end object detector.

Motivation & Objective

• Motivate and validate that modeling relations between objects improves detection performance in deep CNNs.
• Develop a lightweight, in-place object relation module that can be integrated into existing detectors without extra supervision.
• Demonstrate improvements in instance recognition and duplicate removal, enabling end-to-end object detection.
• Provide end-to-end training evidence showing joint optimization of proposal, recognition, and duplicate removal stages.

Proposed method

• Extend attention mechanisms to operate over a set of objects rather than sequence elements.
• Compute an object relation feature by combining a learned appearance-based attention with a geometry-based weight that is translation-invariant.
• Use a multi-relational scheme with multiple relation heads to capture diverse interactions, and aggregate relation features with the input appearance features.
• Replace traditional NMS with a learnable duplicate removal network that reasons over all proposals and their scores, using both rank features and spatial geometry.
• Allow end-to-end training by back-propagating through the relation module into backbone and head networks.

Experimental results

Research questions

• RQ1 Can an attention-like object relation module improve instance recognition when integrated into standard region-based detectors?
• RQ2 Does modeling pairwise object geometry and appearance relations improve duplicate removal beyond traditional NMS/Soft-NMS?
• RQ3 Is end-to-end training feasible and beneficial when the relation module jointly optimizes proposal, recognition, and duplicate removal stages?
• RQ4 How do the number of relations and relation modules affect detection performance and computational cost?

Key findings

• Incorporating the object relation module yields significant gains over a baseline 2fc head, with mAP improving from 29.6 to 31.9 when using a single relation module, and up to 32.5 mAP with additional configurations.
• Using multiple relation heads and relation modules yields further gains, achieving 32.5 mAP with 2fc+RM and 1,1 configuration, and up to 34.1 mAP when combining 2x context and RM in certain setups.
• The geometry weight is shown to be important; replacing it with simpler variants reduces performance, and an optimal design includes a learned geometry embedding that is translation-invariant.
• The proposed duplicate removal network, trained with the relation module, outperforms traditional NMS/Soft-NMS in both accuracy and speed, and enables end-to-end training.
• End-to-end training that jointly optimizes proposal, recognition, and duplicate removal improves recognition accuracy compared to training stages independently.