[Paper Review] RGB-D Salient Object Detection via 3D Convolutional Neural Networks

Qian Chen, Ze Liu+4 more|arXiv (Cornell University)|Jan 25, 2021

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes RD3D, the first 3D CNN-based model for RGB-D salient object detection, which leverages inflated 3D convolutions for early pre-fusion of RGB and depth features in the encoder and employs a 3D decoder with rich back-projection paths (RBPP) and channel-modality attention (CMA) for deep fusion. RD3D outperforms 14 state-of-the-art methods across six benchmark datasets, demonstrating superior accuracy through progressive, end-to-end 3D cross-modal feature integration.

ABSTRACT

RGB-D salient object detection (SOD) recently has attracted increasing research interest and many deep learning methods based on encoder-decoder architectures have emerged. However, most existing RGB-D SOD models conduct feature fusion either in the single encoder or the decoder stage, which hardly guarantees sufficient cross-modal fusion ability. In this paper, we make the first attempt in addressing RGB-D SOD through 3D convolutional neural networks. The proposed model, named RD3D, aims at pre-fusion in the encoder stage and in-depth fusion in the decoder stage to effectively promote the full integration of RGB and depth streams. Specifically, RD3D first conducts pre-fusion across RGB and depth modalities through an inflated 3D encoder, and later provides in-depth feature fusion by designing a 3D decoder equipped with rich back-projection paths (RBPP) for leveraging the extensive aggregation ability of 3D convolutions. With such a progressive fusion strategy involving both the encoder and decoder, effective and thorough interaction between the two modalities can be exploited and boost the detection accuracy. Extensive experiments on six widely used benchmark datasets demonstrate that RD3D performs favorably against 14 state-of-the-art RGB-D SOD approaches in terms of four key evaluation metrics. Our code will be made publicly available: https://github.com/PPOLYpubki/RD3D.

Motivation & Objective

• To address the limited cross-modal fusion capability in existing RGB-D SOD models that fuse features either only in the encoder or decoder.
• To explore the potential of 3D convolutional neural networks for effective, progressive fusion of RGB and depth modalities in salient object detection.
• To design a fully 3D CNN-based framework that enables both pre-fusion in the encoder and explicit, in-depth fusion in the decoder.
• To eliminate the need for complex, dedicated fusion modules by leveraging the inherent feature aggregation capacity of 3D convolutions.
• To validate the effectiveness of rich back-projection paths (RBPP) and channel-modality attention (CMA) in enhancing multi-modal feature learning.

Proposed method

• The model uses an inflated 3D ResNet-based encoder to perform pre-fusion of RGB and depth inputs by treating depth as a temporal dimension, enabling early cross-modal interaction.
• A novel 3D decoder with rich back-projection paths (RBPP) is designed to aggregate multi-level features from the encoder, enhancing spatial and modality-aware representation learning.
• Channel-modality attention (CMA) modules are integrated into the decoder to dynamically recalibrate feature maps based on channel and modality-specific importance.
• The framework employs a progressive fusion strategy: pre-fusion in the 3D encoder and explicit, modality-aware fusion in the 3D decoder, ensuring comprehensive cross-modal integration.
• The model is trained end-to-end using standard loss functions for saliency prediction, with feature maps processed across spatial and temporal (modality) dimensions.
• The architecture is fully 3D, with 3D convolutions applied throughout both encoder and decoder, distinguishing it from 2D-based prior approaches.

Experimental results

Research questions

• RQ1 Can 3D convolutional neural networks effectively enable pre-fusion of RGB and depth features in the encoder for improved salient object detection?
• RQ2 Does a 3D decoder with rich back-projection paths (RBPP) enhance feature aggregation and detection accuracy compared to standard 2D or 3D U-Net decoders?
• RQ3 How does the proposed channel-modality attention (CMA) mechanism improve modality-aware feature learning compared to standard channel attention?
• RQ4 Is a fully 3D CNN-based framework superior to existing 2D-based encoder-decoder models in RGB-D SOD, especially in cross-modal feature integration?
• RQ5 What is the relative contribution of pre-fusion in the encoder versus fusion in the decoder to overall performance?

Key findings

• RD3D achieves state-of-the-art performance on six widely used RGB-D SOD benchmarks, outperforming 14 existing SOTA methods across four evaluation metrics.
• The ablation study confirms that the baseline 3D ResNet model without additional modules (Model-1) already surpasses two recent SOTA models, DANet and JL-DCF, demonstrating the inherent strength of 3D convolutions.
• Removing rich back-projection paths (RBPP) leads to consistent performance degradation, proving their effectiveness in leveraging multi-level, modality-aware features.
• Replacing the proposed CMA modules with standard squeeze-excitation attention reduces performance, confirming the superiority of the channel-modality attention mechanism in the 3D context.
• The full RD3D model with both RBPP and CMA achieves the best results, and ablation shows that combining both components yields substantial performance gains.
• The model maintains comparable inference speed and model size to two-stream networks, indicating that the 3D design does not incur significant computational overhead.