[Paper Review] RGB-T Tracking via Multi-Modal Mutual Prompt Learning

Yang Luo, Xiqing Guo+2 more|arXiv (Cornell University)|Aug 31, 2023

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes Multi-Modal Mutual Prompt Learning Tracker (MPLT), a novel RGB-T tracking framework that enables bidirectional, lightweight feature fusion between visible and thermal modalities via dual-branch Siamese vision transformers enhanced with multi-head attention-based prompters. The method achieves state-of-the-art performance with high inference speed and low computational cost, outperforming existing methods on multiple benchmarks including LasHeR and RGBT210.

ABSTRACT

Object tracking based on the fusion of visible and thermal im-ages, known as RGB-T tracking, has gained increasing atten-tion from researchers in recent years. How to achieve a more comprehensive fusion of information from the two modalities with fewer computational costs has been a problem that re-searchers have been exploring. Recently, with the rise of prompt learning in computer vision, we can better transfer knowledge from visual large models to downstream tasks. Considering the strong complementarity between visible and thermal modalities, we propose a tracking architecture based on mutual prompt learning between the two modalities. We also design a lightweight prompter that incorporates attention mechanisms in two dimensions to transfer information from one modality to the other with lower computational costs, embedding it into each layer of the backbone. Extensive ex-periments have demonstrated that our proposed tracking ar-chitecture is effective and efficient, achieving state-of-the-art performance while maintaining high running speeds.

Motivation & Objective

• Address the challenge of efficient and effective fusion of visible and thermal image modalities in RGB-T tracking.
• Overcome the limitations of unidirectional prompt learning and frozen backbone architectures in reducing computational cost and improving modality interaction.
• Design a lightweight, trainable prompt module that enables bidirectional, multi-level feature exchange during feature extraction.
• Enhance robustness to illumination changes, occlusions, and thermal crossover through online template updating and Kalman filtering.
• Achieve high tracking accuracy with minimal parameter growth and low FLOPS, enabling real-time deployment.

Proposed method

• Extend Vision Transformer (ViT) backbone into a dual-branch Siamese architecture to process RGB and thermal images separately.
• Introduce a Multi-Modal Visual Information Prompter (MVIP) that combines token-level and spatial-level attention to generate modality-specific weights.
• Use the MVIP to adaptively attend to features from one modality and inject them into the other modality’s feature map at each layer via weighted addition.
• Integrate the prompt output with the backbone’s layer output, enabling hierarchical, multi-level fusion across feature maps.
• Apply an online template update strategy based on classification confidence to adapt to appearance variations.
• Incorporate Kalman filtering for prediction box refinement to improve robustness under occlusion and motion blur.

Experimental results

Research questions

• RQ1 Can bidirectional, lightweight prompt learning improve feature fusion in RGB-T tracking compared to unidirectional or direct concatenation?
• RQ2 Does integrating multi-head attention mechanisms in the prompt module enhance cross-modal feature alignment and representation quality?
• RQ3 Can a trainable prompter with minimal parameters achieve performance comparable to or better than frozen-backbone methods while reducing GPU memory usage?
• RQ4 How effective is the proposed online template updating and Kalman filtering refinement in handling appearance variations and occlusions?
• RQ5 To what extent does the mutual prompt learning mechanism improve robustness in challenging scenarios such as low-light, thermal crossover, and fast motion?

Key findings

• MPLT achieves 72.0% PR and 57.1% SR on the LasHeR dataset, outperforming state-of-the-art methods including ViPT, TBSI, and MACFT.
• On the RGBT210 dataset, MPLT achieves 86.2% PR and 63.0% SR, surpassing all compared methods including CAT, DMCNet, and mfDiMP.
• Abllation studies show that removing the MVIP module reduces PR by 6.1% and SR by 5.6%, confirming its critical role in performance.
• The removal of spatial or token attention in MVIP leads to a 3.5–4.7% drop in PR, indicating both attention mechanisms are essential for effective fusion.
• Freezing the backbone reduces performance by 3.8% in PR and 3.1% in SR, but MPLT-F still outperforms ViPT, demonstrating the robustness of the mutual prompt design.
• MPLT uses only 97M trainable parameters and 58.7G FLOPS, significantly lower than TBSI (307M params, 82.2G FLOPS), proving superior efficiency.