[Paper Review] RDSNet: A New Deep Architecture for Reciprocal Object Detection and Instance Segmentation

Shaoru Wang, Yongchao Gong+4 more|arXiv (Cornell University)|Dec 11, 2019

Advanced Neural Network Applications30 references16 citations

TL;DR

RDSNet proposes a unified two-stream deep architecture that jointly optimizes object detection and instance segmentation through reciprocal feature fusion. By alternately refining bounding boxes using pixel-level masks and improving mask quality via object-level cues, RDSNet achieves state-of-the-art accuracy (32.1 mAP at 32 fps) with high-resolution masks and reduced localization errors, outperforming one-stage methods like YOLACT and addressing key limitations of two-stage models such as Mask R-CNN.

ABSTRACT

Object detection and instance segmentation are two fundamental computer vision tasks. They are closely correlated but their relations have not yet been fully explored in most previous work. This paper presents RDSNet, a novel deep architecture for reciprocal object detection and instance segmentation. To reciprocate these two tasks, we design a two-stream structure to learn features on both the object level (i.e., bounding boxes) and the pixel level (i.e., instance masks) jointly. Within this structure, information from the two streams is fused alternately, namely information on the object level introduces the awareness of instance and translation variance to the pixel level, and information on the pixel level refines the localization accuracy of objects on the object level in return. Specifically, a correlation module and a cropping module are proposed to yield instance masks, as well as a mask based boundary refinement module for more accurate bounding boxes. Extensive experimental analyses and comparisons on the COCO dataset demonstrate the effectiveness and efficiency of RDSNet. The source code is available at https://github.com/wangsr126/RDSNet.

Motivation & Objective

• To address the mutual dependency between object detection and instance segmentation, which are often treated as separate tasks despite their strong correlation.
• To overcome limitations in existing methods, such as low mask resolution, high dependency on detection results, and bounding box localization errors.
• To design a unified, end-to-end architecture that leverages object-level and pixel-level features in a reciprocal manner for improved performance.
• To achieve real-time inference speed while maintaining high accuracy, especially for small and large objects.
• To reduce reliance on regression-based bounding box prediction by introducing a mask-based boundary refinement mechanism.

Proposed method

• RDSNet employs a two-stream architecture: an object stream for regression-based detection and a pixel stream for high-resolution instance segmentation using a fully convolutional network (FCN) backbone.
• A correlation module and a cropping module are introduced to transfer instance-awareness and translation-variant features from the object stream to the pixel stream, enabling precise mask generation.
• The mask-based boundary refinement module (MBRM) uses the predicted instance mask to refine bounding box coordinates, reducing localization errors by leveraging the minimum enclosing rectangle of the mask.
• During training, an expand-and-contrast strategy is applied to handle background pixel diversity, and OHEM (Online Hard Example Mining) is used to improve mask quality.
• Features are fused alternately between streams, allowing the object stream to benefit from pixel-level context and the pixel stream to refine object localization.
• The model is trained end-to-end with a multi-task loss combining detection and segmentation objectives, with hyperparameters tuned for optimal boundary refinement.

Experimental results

Research questions

• RQ1 Can joint optimization of object detection and instance segmentation through reciprocal feature learning improve performance on both tasks?
• RQ2 How can mask quality be improved independently of detection box quality, especially for large objects?
• RQ3 To what extent can bounding box localization accuracy be enhanced by using pixel-level mask information instead of relying solely on regression?
• RQ4 What training strategies are most effective for reducing background pixel confusion and improving mask generalization?
• RQ5 How does the proposed MBRM module compare to direct mask-to-box mapping in terms of localization accuracy across object scales?

Key findings

• RDSNet achieves 32.1 mAP on COCO instance segmentation with an input size of 550 and runs at 32 fps, outperforming YOLACT by 1.1 mAP and matching the accuracy of slower two-stage models.
• The cropping module improves mAP by 1.1 points over linear combination (LC) and 1.9 points over YOLACT when combined with OHEM, demonstrating its effectiveness in mask generation.
• The MBRM module improves bounding box localization, especially for large objects, with a 3.2 mAP increase on large objects (AP_L^bb) compared to baseline, while maintaining stable performance across scales.
• The model reduces localization errors significantly: MBRM improves AP_bb by 1.3 points on average, with the largest gain on large objects (+3.2 mAP), showing the value of mask-guided refinement.
• The ablation study confirms that OHEM and proper data augmentation (e.g., expanding boxes during training) are crucial for performance, with OHEM alone yielding a 1.9 mAP gain over YOLACT.
• The inference speed is nearly identical to the base detector (10.9 fps), indicating minimal overhead from the additional modules, and the model is easily adaptable to faster detectors.