[Paper Review] Does Thermal Really Always Matter for RGB-T Salient Object Detection?

Runmin Cong, Kepu Zhang+5 more|arXiv (Cornell University)|Oct 9, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes TNet, a novel RGB-T salient object detection network that adaptively regulates thermal modality usage based on scene illumination via a global illumination estimation module. It employs a semantic constraint provider in encoding and a two-stage localization/complementation module in decoding to enhance cross-modality interaction, achieving state-of-the-art performance on three benchmark datasets with a maximum F-measure of 0.895 on VT5000.

ABSTRACT

In recent years, RGB-T salient object detection (SOD) has attracted continuous attention, which makes it possible to identify salient objects in environments such as low light by introducing thermal image. However, most of the existing RGB-T SOD models focus on how to perform cross-modality feature fusion, ignoring whether thermal image is really always matter in SOD task. Starting from the definition and nature of this task, this paper rethinks the connotation of thermal modality, and proposes a network named TNet to solve the RGB-T SOD task. In this paper, we introduce a global illumination estimation module to predict the global illuminance score of the image, so as to regulate the role played by the two modalities. In addition, considering the role of thermal modality, we set up different cross-modality interaction mechanisms in the encoding phase and the decoding phase. On the one hand, we introduce a semantic constraint provider to enrich the semantics of thermal images in the encoding phase, which makes thermal modality more suitable for the SOD task. On the other hand, we introduce a two-stage localization and complementation module in the decoding phase to transfer object localization cue and internal integrity cue in thermal features to the RGB modality. Extensive experiments on three datasets show that the proposed TNet achieves competitive performance compared with 20 state-of-the-art methods.

Motivation & Objective

• To investigate whether thermal imaging always improves RGB-T salient object detection, challenging the assumption that thermal data is universally beneficial.
• To address the suboptimal performance of existing models that directly transfer RGB-D SOD methods to RGB-T SOD, which misaligns with the physical and perceptual differences between depth and thermal modalities.
• To design a network that dynamically regulates the contribution of thermal and RGB modalities based on scene brightness, improving robustness in low-light conditions.
• To enhance cross-modality feature interaction by introducing task-specific modules in both encoding and decoding stages for semantic enrichment and structural complementation.
• To demonstrate that thermal modality's value is context-dependent and should not be uniformly applied, especially in scenes where thermal and RGB saliency do not align.

Proposed method

• A global illumination estimation (GIE) module predicts the global illuminance score of an RGB image to dynamically regulate the fusion strategy between RGB and thermal features.
• In the encoding phase, a semantic constraint provider (SCP) enriches thermal features by injecting semantic masks derived from high-level RGB features, improving thermal feature relevance for SOD.
• In the decoding phase, a two-stage localization and complementation (LC) module transfers localization cues and structural integrity from thermal features to RGB features via adaptive skip connections.
• The LC module first uses thermal features to guide object localization in RGB features, then applies a weighted fusion mechanism based on the GIE score to complement RGB features with thermal details.
• The adaptive skip connection mechanism uses the global illuminance score to compute attention weights, enabling dynamic, context-aware fusion instead of fixed addition.
• The full TNet architecture integrates GIE, SCP, and LC modules into an end-to-end trainable framework for joint RGB and thermal feature learning.

Experimental results

Research questions

• RQ1 Does the thermal modality always enhance performance in RGB-T salient object detection, or are there scenarios where it degrades performance?
• RQ2 How can the role of thermal modality be adaptively regulated based on scene illumination conditions?
• RQ3 What are the most effective cross-modality interaction mechanisms in encoding and decoding stages for RGB-T SOD?
• RQ4 Can semantic enhancement of thermal features and structural complementation from thermal to RGB improve detection accuracy and completeness?
• RQ5 How does the proposed adaptive fusion mechanism compare to fixed or direct fusion strategies in terms of robustness and performance?

Key findings

• The proposed TNet achieves a maximum F-measure of 0.895 on the VT5000 dataset, outperforming 20 state-of-the-art methods.
• Ablation studies show that removing the localization guidance stage reduces the F-measure to 0.885, and removing the complementation stage drops it to 0.873, confirming both stages' importance.
• Replacing adaptive skip connections with direct addition reduces the F-measure from 0.895 to 0.882 and increases MAE from 0.033 to 0.037, demonstrating the superiority of the adaptive fusion mechanism.
• The global illumination estimation module effectively regulates modality interaction, with performance degradation observed when it is removed.
• Failure cases reveal limitations in detecting small or low-contrast, elongated objects, indicating room for improvement with stronger feature extractors like Swin Transformer.
• The current RGB-T SOD datasets are biased toward daytime scenes, highlighting the need for more low-light, nighttime-focused benchmarks.