[Paper Review] RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentation

Jindong Jiang, Lunan Zheng+2 more|arXiv (Cornell University)|Jun 4, 2018

Remote Sensing and LiDAR Applications37 references181 citations

TL;DR

RedNet introduces a residual encoder-decoder with RGB-D fusion and pyramid supervision for indoor semantic segmentation, achieving 47.8% mIoU on SUN RGB-D with a ResNet-50 backbone.

ABSTRACT

Indoor semantic segmentation has always been a difficult task in computer vision. In this paper, we propose an RGB-D residual encoder-decoder architecture, named RedNet, for indoor RGB-D semantic segmentation. In RedNet, the residual module is applied to both the encoder and decoder as the basic building block, and the skip-connection is used to bypass the spatial feature between the encoder and decoder. In order to incorporate the depth information of the scene, a fusion structure is constructed, which makes inference on RGB image and depth image separately, and fuses their features over several layers. In order to efficiently optimize the network's parameters, we propose a `pyramid supervision' training scheme, which applies supervised learning over different layers in the decoder, to cope with the problem of gradients vanishing. Experiment results show that the proposed RedNet(ResNet-50) achieves a state-of-the-art mIoU accuracy of 47.8% on the SUN RGB-D benchmark dataset.

Motivation & Objective

• Improve indoor RGB-D semantic segmentation using a deep encoder-decoder architecture.
• Incorporate depth information via a dual-branch RGB-D fusion strategy.
• Mitigate gradient vanishing with pyramid supervision across decoder layers.
• Enable deeper networks by using residual blocks in both encoder and decoder.
• Evaluate RedNet on SUN RGB-D to benchmark performance.

Proposed method

• Use a dual-branch RGB and Depth encoder (ResNet-50 or ResNet-34) with residual blocks.
• Fuse depth features into the RGB branch via element-wise summation at multiple layers.
• Implement an upsampling residual unit in the decoder to recover full resolution.
• Apply pyramid supervision by adding side outputs from multiple decoder layers with corresponding losses.
• Train with weighted cross-entropy using median frequency balancing and ImageNet-pretrained encoders.
• Optionally use agent layers to reduce memory when using ResNet-50.

Experimental results

Research questions

• RQ1 Can a residual encoder-decoder with RGB-D fusion outperform existing indoor RGB-D segmentation models?
• RQ2 Does depth fusion at multiple encoder layers improve segmentation accuracy?
• RQ3 Does pyramid supervision improve optimization and final performance?

Key findings

• RedNet-34 achieves 46.8 mIoU, 81.3 pixel accuracy, and 60.3 mean accuracy on SUN RGB-D when using pyramid supervision with ResNet-50.
• RedNet-50 achieves 47.8 mIoU, 81.3 pixel accuracy, and 60.3 mean accuracy on SUN RGB-D with pyramid supervision.
• Without pyramid supervision, RedNet-34 attains 45.0 mIoU, and RedNet-50 attains 46.0 mIoU.
• RedNet-50 with pyramid supervision outperforms its non-pyramid counterpart by about 1.8 mIoU (47.8 vs 46.0).
• Overall, RedNet variants outperform several prior RGB-D semantic segmentation methods on SUN RGB-D.