[Paper Review] Triply Supervised Decoder Networks for Joint Detection and Segmentation

Jiale Cao, Yanwei Pang+1 more|arXiv (Cornell University)|Sep 25, 2018

Advanced Neural Network Applications42 references4 citations

TL;DR

This paper proposes TripleNet, a novel fully convolutional network that jointly performs object detection and semantic segmentation by applying triple supervision—detection-oriented, class-aware segmentation, and class-agnostic segmentation—on each decoder layer. The method enhances feature learning through lightweight modules (inner-connected module and attention skip-layer fusion), achieving state-of-the-art performance on VOC2007 and VOC2012 without increasing inference-time computational cost.

ABSTRACT

Joint object detection and semantic segmentation can be applied to many fields, such as self-driving cars and unmanned surface vessels. An initial and important progress towards this goal has been achieved by simply sharing the deep convolutional features for the two tasks. However, this simple scheme is unable to make full use of the fact that detection and segmentation are mutually beneficial. To overcome this drawback, we propose a framework called TripleNet where triple supervisions including detection-oriented supervision, class-aware segmentation supervision, and class-agnostic segmentation supervision are imposed on each layer of the decoder network. Class-agnostic segmentation supervision provides an objectness prior knowledge for both semantic segmentation and object detection. Besides the three types of supervisions, two light-weight modules (i.e., inner-connected module and attention skip-layer fusion) are also incorporated into each layer of the decoder. In the proposed framework, detection and segmentation can sufficiently boost each other. Moreover, class-agnostic and class-aware segmentation on each decoder layer are not performed at the test stage. Therefore, no extra computational costs are introduced at the test stage. Experimental results on the VOC2007 and VOC2012 datasets demonstrate that the proposed TripleNet is able to improve both the detection and segmentation accuracies without adding extra computational costs.

Motivation & Objective

• To address the limited mutual benefit between object detection and semantic segmentation in existing joint learning frameworks.
• To improve performance in both detection and segmentation tasks by leveraging complementary supervisory signals.
• To design a lightweight architecture that avoids introducing extra inference cost despite multi-task supervision.
• To explore the effectiveness of class-agnostic segmentation as a prior for both detection and segmentation tasks.

Proposed method

• Imposes three types of supervision—detection-oriented, class-aware segmentation, and class-agnostic segmentation—on each decoder layer to enhance feature representation.
• Introduces an inner-connected module to strengthen feature learning within each decoder layer by connecting features across channels.
• Employs attention-based skip-layer fusion to dynamically weight and fuse features from encoder and decoder paths, improving feature quality.
• Uses a multi-scale decoder architecture where each layer predicts objects of different scales, enabling multi-scale detection.
• Applies class-agnostic segmentation supervision as a prior to guide both detection and segmentation, improving objectness estimation.
• Performs class-aware and class-agnostic segmentation only during training; these modules are removed at test time to avoid computational overhead.

Experimental results

Research questions

• RQ1 Can joint supervision across detection and segmentation tasks improve performance in both tasks simultaneously?
• RQ2 Does class-agnostic segmentation supervision act as a useful prior for improving detection and segmentation accuracy?
• RQ3 Can lightweight modules like inner-connected and attention-based skip-layer fusion enhance feature learning without increasing inference cost?
• RQ4 Is it possible to achieve state-of-the-art performance in both detection and segmentation without adding computational cost at inference time?
• RQ5 How does the proposed TripleNet compare to existing joint detection and segmentation methods in terms of accuracy and efficiency?

Key findings

• TripleNet achieves a mean average precision (mAP) of 82.7% on the VOC2007 test set, outperforming all state-of-the-art methods including SSD, DSSD, and BlitzNet.
• On the VOC2012 dataset, TripleNet achieves a mean Intersection over Union (mIoU) of 82.9%, surpassing BlitzNet by 7.3 percentage points.
• The method improves detection mAP by 2.0% and segmentation mIoU by 7.3% compared to BlitzNet, demonstrating strong performance gains.
• The proposed TripleNet achieves state-of-the-art performance on both detection and segmentation tasks without introducing any additional computational cost at inference time.
• The ablation study confirms that class-agnostic segmentation supervision significantly boosts performance, especially in object detection.
• The attention skip-layer fusion and inner-connected modules contribute to improved feature representation, as shown by ablation results on both datasets.