[Paper Review] Accurate RGB-D Salient Object Detection via Collaborative Learning

Wei Ji, Jingjing Li+3 more|arXiv (Cornell University)|Jul 23, 2020

Visual Attention and Saliency Detection69 references14 citations

TL;DR

This paper proposes CoNet, a collaborative learning framework for accurate RGB-D salient object detection that jointly leverages edge, depth, and saliency features in a mutually beneficial manner. By integrating depth and saliency learning into high-level feature learning without requiring an extra depth network or depth input during inference, the method achieves state-of-the-art performance with a lightweight, fast 34 FPS inference speed and sharp boundary preservation.

ABSTRACT

Benefiting from the spatial cues embedded in depth images, recent progress on RGB-D saliency detection shows impressive ability on some challenge scenarios. However, there are still two limitations. One hand is that the pooling and upsampling operations in FCNs might cause blur object boundaries. On the other hand, using an additional depth-network to extract depth features might lead to high computation and storage cost. The reliance on depth inputs during testing also limits the practical applications of current RGB-D models. In this paper, we propose a novel collaborative learning framework where edge, depth and saliency are leveraged in a more efficient way, which solves those problems tactfully. The explicitly extracted edge information goes together with saliency to give more emphasis to the salient regions and object boundaries. Depth and saliency learning is innovatively integrated into the high-level feature learning process in a mutual-benefit manner. This strategy enables the network to be free of using extra depth networks and depth inputs to make inference. To this end, it makes our model more lightweight, faster and more versatile. Experiment results on seven benchmark datasets show its superior performance.

Motivation & Objective

• Address the limitations of existing RGB-D saliency models, including blurred object boundaries due to pooling/upsampling in FCNs and high computational cost from separate depth networks.
• Overcome the dependency on depth inputs during inference, which restricts practical deployment of RGB-D models.
• Improve boundary accuracy and model efficiency by explicitly integrating edge information and co-learning depth and saliency representations.
• Develop a unified, lightweight framework that maintains high performance without requiring additional depth features or inputs at test time.

Proposed method

• Introduce a collaborative learning framework (CoNet) with three mutually beneficial collaborators: edge detection, coarse saliency detection, and depth estimation.
• Use a dedicated edge collaborator to extract and enhance edge features from low-level features, improving boundary localization in saliency maps.
• Innovatively integrate depth and saliency learning into high-level feature learning, enabling the network to infer depth-related semantics without using depth images during inference.
• Design a knowledge collector (tutor) to aggregate and transfer knowledge from individual collaborators to the main saliency head, enhancing feature representation.
• Train the model end-to-end using multi-task learning, where edge, depth, and saliency supervision are jointly optimized.
• Eliminate the need for an auxiliary depth network by learning depth-aware representations directly from RGB features, reducing model size and inference time.

Experimental results

Research questions

• RQ1 Can edge supervision improve boundary accuracy in RGB-D saliency detection without increasing model complexity?
• RQ2 Can depth and saliency learning be jointly optimized in high-level features to eliminate the need for separate depth networks?
• RQ3 Can a collaborative learning framework achieve state-of-the-art performance while being faster and more lightweight than existing RGB-D models?
• RQ4 Is it possible to maintain high accuracy in saliency detection without requiring depth input during inference?
• RQ5 How does the proposed method compare to both RGB-only and RGB-D state-of-the-art models in terms of speed, accuracy, and model efficiency?

Key findings

• CoNet achieves state-of-the-art performance on seven benchmark datasets, with $F_{\beta}^{w}$ scores of 0.856 (NJUD), 0.850 (NLPR), and 0.871 (DUT-RS), outperforming 22 SOTA methods.
• The model achieves 34 FPS inference speed, a 55% improvement over the next-best RGB-D model (DMRA), making it suitable for real-time applications.
• On all seven datasets, CoNet achieves the lowest MAE values, with 0.031 on DUT-RS and 0.031 on NLPR, indicating high precision in saliency prediction.
• The model is significantly more efficient, with only 167.6 MB model size, outperforming larger models like TANet (951.9 MB) and MPCI (929.7 MB) in both speed and accuracy.
• Visual results show superior boundary preservation and accuracy on complex scenes with low contrast or multiple objects, thanks to edge and depth-aware feature learning.
• Quantitative comparisons confirm that CoNet performs comparably to top RGB-only methods (e.g., CPD, PoolNet) while leveraging depth information more effectively.