[Paper Review] Self-Supervised Representation Learning for RGB-D Salient Object Detection.

Xiaoqi Zhao, Youwei Pang+3 more|arXiv (Cornell University)|Jan 29, 2021

Visual Attention and Saliency Detection62 references4 citations

TL;DR

This paper proposes a self-supervised representation learning framework for RGB-D salient object detection that pre-trains on unannotated RGB-D data using two pretext tasks—cross-modal auto-encoder and depth-contour estimation—achieving performance comparable to ImageNet-pretrained models, even with only 6,392 unannotated images.

ABSTRACT

Existing CNNs-Based RGB-D Salient Object Detection (SOD) networks are all required to be pre-trained on the ImageNet to learn the hierarchy features which can help to provide a good initialization. However, the collection and annotation of large-scale datasets are time-consuming and expensive. In this paper, we utilize Self-Supervised Representation Learning (SSL) to design two pretext tasks: the cross-modal auto-encoder and the depth-contour estimation. Our pretext tasks require only a few and unlabeled RGB-D datasets to perform pre-training, which makes the network capture rich semantic contexts and reduce the gap between two modalities, thereby providing an effective initialization for the downstream task. In addition, for the inherent problem of cross-modal fusion in RGB-D SOD, we propose a consistency-difference aggregation (CDA) module that splits a single feature fusion into multi-path fusion to achieve an adequate perception of consistent and differential information. The CDA module is general and suitable for both cross-modal and cross-level feature fusion. Extensive experiments on six benchmark RGB-D SOD datasets, our model pre-trained on the RGB-D dataset ($6,392$ without any annotations) can perform favorably against most state-of-the-art RGB-D methods pre-trained on ImageNet ($1,280,000$ with image-level annotations).

Motivation & Objective

• Address the high cost and effort of large-scale annotated datasets for pre-training in RGB-D salient object detection.
• Reduce reliance on ImageNet pre-training by leveraging self-supervised learning on unannotated RGB-D data.
• Improve cross-modal feature alignment and semantic understanding in RGB-D fusion through pretext tasks.
• Overcome the inherent challenge of cross-modal fusion in RGB-D SOD by designing a general-purpose fusion module.
• Achieve state-of-the-art performance with minimal supervision by pre-training on a small, unannotated RGB-D dataset.

Proposed method

• Design a cross-modal auto-encoder pretext task that reconstructs RGB and depth modality features from a shared latent representation, encouraging modality alignment.
• Introduce a depth-contour estimation pretext task that predicts depth edge maps from RGB features, enhancing geometric and structural understanding.
• Propose a consistency-difference aggregation (CDA) module that decomposes feature fusion into consistent and differential paths for improved cross-modal and cross-level feature interaction.
• Train the network end-to-end using the two pretext tasks on a small, unannotated RGB-D dataset of 6,392 images to learn rich hierarchical representations.
• Fine-tune the pre-trained model on downstream RGB-D SOD tasks using only bounding box or segmentation annotations.
• Apply the CDA module to both cross-modal and cross-level feature fusion, enabling effective perception of shared and unique information across modalities and feature levels.

Experimental results

Research questions

• RQ1 Can self-supervised pretext tasks on unannotated RGB-D data effectively replace ImageNet pre-training for RGB-D salient object detection?
• RQ2 How well can cross-modal auto-encoder and depth-contour estimation pretext tasks capture semantic and structural information in RGB-D data?
• RQ3 To what extent does the consistency-difference aggregation (CDA) module improve cross-modal feature fusion in RGB-D SOD?
• RQ4 Can a model pre-trained on only 6,392 unannotated RGB-D images achieve performance competitive with ImageNet-pretrained models on benchmark datasets?
• RQ5 How generalizable is the CDA module across different fusion scenarios, such as cross-modal and cross-level feature fusion?

Key findings

• The proposed self-supervised pre-training on 6,392 unannotated RGB-D images achieves performance comparable to state-of-the-art methods pre-trained on ImageNet with 1,280,000 image-level annotations.
• The cross-modal auto-encoder and depth-contour estimation pretext tasks significantly improve feature representation quality and modality alignment.
• The consistency-difference aggregation (CDA) module enhances feature fusion by explicitly modeling both consistent and differential information across modalities and feature levels.
• The model outperforms or matches most existing RGB-D SOD methods across six benchmark datasets, even without any image-level annotations during pre-training.
• The ablation study confirms that both pretext tasks contribute to improved performance, with depth-contour estimation being particularly effective for edge and contour understanding.
• The CDA module generalizes well to both cross-modal and cross-level fusion, demonstrating consistent gains in accuracy and robustness.