[Paper Review] Duality-Gated Mutual Condition Network for RGBT Tracking

Andong Lu, Cun Qian+3 more|arXiv (Cornell University)|Nov 14, 2020

Retinal Imaging and Analysis4 citations

TL;DR

This paper proposes a Duality-Gated Mutual Condition Network (DMCNet) for RGBT tracking that enhances feature representation by using discriminative features from one modality as conditional guidance for the other, while suppressing noise via a duality-gated mechanism. The method achieves state-of-the-art performance on four benchmark datasets by effectively leveraging low-quality modalities and incorporating an optical flow-based resampling strategy for robustness to sudden camera motion.

ABSTRACT

Low-quality modalities contain not only a lot of noisy information but also some discriminative features in RGBT tracking. However, the potentials of low-quality modalities are not well explored in existing RGBT tracking algorithms. In this work, we propose a novel duality-gated mutual condition network to fully exploit the discriminative information of all modalities while suppressing the effects of data noise. In specific, we design a mutual condition module, which takes the discriminative information of a modality as the condition to guide feature learning of target appearance in another modality. Such module can effectively enhance target representations of all modalities even in the presence of low-quality modalities. To improve the quality of conditions and further reduce data noise, we propose a duality-gated mechanism and integrate it into the mutual condition module. To deal with the tracking failure caused by sudden camera motion, which often occurs in RGBT tracking, we design a resampling strategy based on optical flow algorithms. It does not increase much computational cost since we perform optical flow calculation only when the model prediction is unreliable and then execute resampling when the sudden camera motion is detected. Extensive experiments on four RGBT tracking benchmark datasets show that our method performs favorably against the state-of-the-art tracking algorithms

Motivation & Objective

• To address the underutilization of discriminative features in low-quality RGB and thermal modalities during tracking.
• To suppress noise from unreliable modalities without discarding potentially useful information.
• To improve tracking robustness under sudden camera motion, a common failure case in RGBT tracking.
• To enable adaptive, bidirectional feature modulation between RGB and thermal streams through conditional feature learning.
• To reduce computational cost while maintaining high accuracy through intelligent resampling triggered only when needed.

Proposed method

• Introduces a mutual condition module that uses discriminative features from one modality as conditional context to guide feature learning in the other modality.
• Employs multi-scale convolutional layers within the mutual condition module to enhance feature representation and capture spatial context.
• Designs a duality-gated mechanism with two parallel gates to refine the conditional features, improving condition quality and reducing noise propagation.
• Incorporates a resampling strategy based on optical flow that activates only when model prediction confidence drops, minimizing computational overhead.
• Uses a bidirectional feature modulation scheme where both modalities condition each other’s feature learning, enabling robust cross-modal representation.
• Applies shifting operations in the conditional module to align features across modalities and improve feature fusion efficiency.

Figure 1: Illustration of the effectiveness of our duality-gated mutual condition network (DMCNet) against a typical tracker MANet [ 12 ] on four examples in soccer , elecbike10 , car10 and soccer2 respectively. Herein, the first row represents the input RGB and thermal frames, and the second row de

Experimental results

Research questions

• RQ1 Can discriminative features from low-quality modalities be effectively leveraged to improve RGBT tracking performance?
• RQ2 How can noise in low-quality modalities be suppressed while preserving useful discriminative signals?
• RQ3 What is the impact of bidirectional conditional feature learning on tracking robustness and accuracy?
• RQ4 Can a lightweight, adaptive resampling mechanism improve tracking under sudden camera motion without increasing computational cost?
• RQ5 How do individual components of the duality-gated mutual condition module contribute to overall performance?

Key findings

• DMCNet achieves state-of-the-art performance on four RGBT tracking benchmarks, with PR/SR scores of 0.839/0.593 on RGBT234 and 0.909/0.733 on GTOT.
• The ablation study confirms that the duality-gated mutual condition module contributes most to performance gain, while the resampling strategy reduces time cost by avoiding unnecessary updates.
• Removing the duality-gated mechanism (DMC-w/o-gate) reduces PR by 0.012 on RGBT234, demonstrating its critical role in noise suppression.
• The multi-scale convolution in the DMC module has the largest impact on speed and performance, improving PR by 0.009 compared to the variant without it.
• The resampling strategy is inactive on GTOT due to fixed cameras, confirming its targeted use in dynamic motion scenarios.
• Replacing the DMC module with FiLM yields lower performance (PR: 0.812 vs. 0.820), proving the superiority of the proposed duality-gated design.

Figure 2: Overall network architecture of our DMCNet. Herein, $\oplus$ , $\copyright$ denote the operations of element-wise addition and concatenation respectively.