[Paper Review] TriTransNet: RGB-D Salient Object Detection with a Triplet Transformer Embedding Network

Zhengyi Liu, Yuan Wang+3 more|arXiv (Cornell University)|Aug 9, 2021

Visual Attention and Saliency Detection89 references18 citations

TL;DR

TriTransNet proposes a novel triplet transformer embedding module (TTEM) that enhances multi-level features in an RGB-D salient object detection network by leveraging three shared-weight transformer encoders to model long-range dependencies across high-level features. The method achieves state-of-the-art performance on benchmark datasets by integrating a three-stream decoder, scale adjustment, and a depth purification module for effective multi-modal fusion.

ABSTRACT

Salient object detection is the pixel-level dense prediction task which can highlight the prominent object in the scene. Recently U-Net framework is widely used, and continuous convolution and pooling operations generate multi-level features which are complementary with each other. In view of the more contribution of high-level features for the performance, we propose a triplet transformer embedding module to enhance them by learning long-range dependencies across layers. It is the first to use three transformer encoders with shared weights to enhance multi-level features. By further designing scale adjustment module to process the input, devising three-stream decoder to process the output and attaching depth features to color features for the multi-modal fusion, the proposed triplet transformer embedding network (TriTransNet) achieves the state-of-the-art performance in RGB-D salient object detection, and pushes the performance to a new level. Experimental results demonstrate the effectiveness of the proposed modules and the competition of TriTransNet.

Motivation & Objective

• To improve salient object detection in RGB-D images by enhancing high-level features using self-attention mechanisms.
• To address the limitation of existing U-Net frameworks in capturing long-range dependencies across feature levels.
• To integrate depth information effectively by designing a depth purification module that reduces noise from low-quality depth maps.
• To achieve superior performance through a three-stream decoder that fuses multi-level features efficiently.
• To demonstrate the effectiveness of shared-weight triplet transformers in enhancing feature representation with minimal computational overhead.

Proposed method

• A triplet transformer embedding module (TTEM) is proposed, consisting of three standard transformer encoders with shared weights to model long-range dependencies across high-level features.
• The TTEM processes multi-level features after they are resized to the same spatial resolution via a transition layer and progressive upsampling fusion.
• A scale adjustment module normalizes input features to improve training stability and feature consistency across levels.
• A three-stream decoder is designed to fuse the enhanced features from TTEM with low-level features, enabling effective multi-scale feature integration.
• A depth purification module (DPM) is introduced to refine depth features before fusing them with RGB features, reducing noise and improving modality alignment.
• Multi-modal fusion is achieved by concatenating or element-wise adding refined depth features with RGB features at multiple stages of the encoder.

Experimental results

Research questions

• RQ1 Can shared-weight triplet transformers effectively enhance high-level features in a U-Net-based salient object detection framework?
• RQ2 How does modeling long-range dependencies across multi-level features improve detection performance?
• RQ3 What is the impact of depth purification on RGB-D salient object detection, especially when depth maps are noisy?
• RQ4 How does the three-stream decoder compare to single-stream decoders in fusing multi-level features?
• RQ5 Can the proposed modules be integrated into existing U-Net architectures with minimal computational cost while achieving SOTA performance?

Key findings

• TriTransNet achieves state-of-the-art performance on four major RGB-D saliency benchmarks: NLPR, NJUD2K, SIP, and STERE, with an S-measure of 0.928 on NLPR and 0.927 on STERE.
• The ablation study confirms that the three-stream decoder significantly improves performance, outperforming a single-stream decoder by 0.005 in S-measure on NLPR and 0.004 on STERE.
• The depth purification module (DPM) improves performance over both baseline element-wise addition and the DEM module from BBS, achieving a 0.005 higher S-measure on NLPR compared to the baseline.
• The triplet transformer embedding module (TTEM) with shared weights leads to a 0.005 increase in Fβ-score on NLPR compared to models without TTEM, demonstrating its effectiveness in feature enhancement.
• The model achieves a mean absolute error (MAE) of 0.020 on NLPR and 0.033 on STERE, indicating high precision in pixel-level prediction.
• The proposed method reduces MAE by 0.003 compared to the baseline on the NLPR dataset, confirming improved localization accuracy.