[Paper Review] MTFNet: Mutual-Transformer Fusion Network for RGB-D Salient Object Detection.

Xixi Wang, Bo Jiang+2 more|arXiv (Cornell University)|Dec 2, 2021

Visual Attention and Saliency Detection48 references6 citations

TL;DR

MTFNet is a novel RGB-D salient object detection network that introduces a Focal Feature Extractor (FFE) for accurate feature learning and a Mutual-Transformer Fusion (MTF) module for simultaneous intra- and inter-modality feature interaction. It achieves state-of-the-art performance across six public benchmarks by effectively leveraging complementary RGB and depth information through joint modality learning.

ABSTRACT

Salient object detection (SOD) on RGB-D images is an active problem in computer vision. The main challenges for RGB-D SOD problem are how to 1) extract the accurate features for RGB and Depth image data with clutter background or poor image quality and 2) explore the complementary information between RGB and Depth image data. To address these challenges, we propose a novel Mutual-Transformer Fusion Network (MTFNet) for RGB-D SOD. MTFNet contains two main modules, $i.e.$, Focal Feature Extractor (FFE) and Mutual-Transformer Fusion (MTF). FFE aims to extract the more accurate CNN features for RGB and Depth images by introducing a novel pixel-level focal regularization to guide CNN feature extractor. MTF is designed to deeply exploit the multi-modal interaction between RGB and Depth images on both coarse and fine scales. The main benefit of MTF is that it conducts the learning of intra-modality and inter-modality simultaneously and thus can achieve communication across different modalities more directly and sufficiently. Comprehensive experimental results on six public benchmarks demonstrate the superiority of our proposed MTFNet.

Motivation & Objective

• Address the challenge of extracting accurate features from RGB and depth images under cluttered backgrounds or poor image quality.
• Exploit complementary information between RGB and depth modalities to improve salient object detection.
• Enable direct and sufficient cross-modality communication by jointly learning intra- and inter-modality representations.
• Develop a unified framework that enhances feature representation through multi-scale fusion of RGB and depth data.

Proposed method

• Introduce a Focal Feature Extractor (FFE) that applies pixel-level focal regularization to guide CNN-based feature extraction for both RGB and depth images.
• Design a Mutual-Transformer Fusion (MTF) module that performs joint learning of intra-modality and inter-modality features across coarse and fine spatial scales.
• Enable bidirectional feature communication between RGB and depth branches through self-attention mechanisms in the MTF module.
• Use multi-scale feature fusion to enhance representation learning at both global and local levels in the feature hierarchy.
• Train the network end-to-end with a supervised loss to optimize saliency prediction performance.
• Leverage the mutual attention mechanism in MTF to dynamically weigh and fuse features based on modality-specific and cross-modal dependencies.

Experimental results

Research questions

• RQ1 How can we improve feature extraction accuracy for RGB and depth images under challenging conditions such as cluttered backgrounds or low image quality?
• RQ2 To what extent can joint intra- and inter-modality learning enhance feature representation in RGB-D salient object detection?
• RQ3 Can a mutual transformer architecture effectively exploit complementary information between RGB and depth modalities at multiple spatial scales?
• RQ4 How does the proposed MTFNet compare to existing state-of-the-art methods on standard RGB-D SOD benchmarks?

Key findings

• MTFNet achieves state-of-the-art performance on six public RGB-D salient object detection benchmarks.
• The Focal Feature Extractor (FFE) significantly improves feature quality by applying pixel-level regularization to CNN-based feature learning.
• The Mutual-Transformer Fusion (MTF) module enables more effective and direct cross-modality communication than prior fusion methods.
• Joint learning of intra- and inter-modality features in MTF leads to superior representation learning and detection accuracy.
• The model demonstrates strong generalization across diverse datasets with varying image quality and background complexity.
• Comprehensive ablation studies confirm the effectiveness of both FFE and MTF components in boosting performance.