[Paper Review] SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding

Wenrui Li, Xiaopeng Hong+2 more|arXiv (Cornell University)|Apr 1, 2024

Visual Attention and Saliency Detection4 citations

TL;DR

SpikeMba proposes a novel multi-modal spiking saliency Mamba network for temporal video grounding, integrating spiking neural networks (SNNs) for dynamic saliency proposal detection and learnable relevant slots to enhance long-sequence memory. The model achieves state-of-the-art performance, with 85.6% R1 and 75.5% mAP on the QVHighlights test set, outperforming prior methods through improved contextual reasoning and saliency localization.

ABSTRACT

Temporal video grounding (TVG) is a critical task in video content understanding, requiring precise alignment between video content and natural language instructions. Despite significant advancements, existing methods face challenges in managing confidence bias towards salient objects and capturing long-term dependencies in video sequences. To address these issues, we introduce SpikeMba: a multi-modal spiking saliency mamba for temporal video grounding. Our approach integrates Spiking Neural Networks (SNNs) with state space models (SSMs) to leverage their unique advantages in handling different aspects of the task. Specifically, we use SNNs to develop a spiking saliency detector that generates the proposal set. The detector emits spike signals when the input signal exceeds a predefined threshold, resulting in a dynamic and binary saliency proposal set. To enhance the model's capability to retain and infer contextual information, we introduce relevant slots which learnable tensors that encode prior knowledge. These slots work with the contextual moment reasoner to maintain a balance between preserving contextual information and exploring semantic relevance dynamically. The SSMs facilitate selective information propagation, addressing the challenge of long-term dependency in video content. By combining SNNs for proposal generation and SSMs for effective contextual reasoning, SpikeMba addresses confidence bias and long-term dependencies, thereby significantly enhancing fine-grained multimodal relationship capture. Our experiments demonstrate the effectiveness of SpikeMba, which consistently outperforms state-of-the-art methods across mainstream benchmarks.

Motivation & Objective

• Address the confidence bias toward salient objects in video grounding, where models overlook subtle but semantically relevant actions.
• Overcome the limitations of long video sequences by enhancing memory capacity and modeling long-term dependencies effectively.
• Improve multimodal feature integration by leveraging spiking neural networks to detect fine-grained saliency proposals.
• Balance contextual information preservation and semantic relevance exploration through a contextual moment reasoner with adaptive relevant slots.
• Reduce computational cost while maintaining high performance in temporal video grounding using efficient state space models (SSMs).

Proposed method

• Introduce a spiking saliency detector based on SNNs that generates binary spike sequences via a threshold mechanism, enabling dynamic, time-step-adaptive proposal generation.
• Propose relevant slots—learnable tensors that encode prior knowledge and selectively represent multi-level multimodal features to enhance memory for long videos.
• Design a contextual moment reasoner (CMR) that uses relevant slots to balance contextual preservation and semantic relevance exploration.
• Employ a multi-modal relevant Mamba block based on state space models (SSMs) to model long-range dependencies with linear complexity.
• Integrate contrastive loss and saliency proposal loss to refine feature representations and improve proposal accuracy.
• Use dynamic parameterization in SSMs to adaptively propagate or forget information based on input, enhancing responsiveness to relevant video segments.

Figure 1 : An illustration of the challenges in video grounding task (left) and the overall architecture of our proposed model (right). On the left, a scenario underscores the problem of confidence bias towards salient objects, with models overemphasizing dramatic changes. On the right, we employ th

Experimental results

Research questions

• RQ1 Can spiking neural networks effectively detect fine-grained saliency proposals in multimodal video grounding tasks?
• RQ2 How do learnable relevant slots improve memory capacity and contextual understanding in long video sequences?
• RQ3 To what extent does the contextual moment reasoner balance semantic relevance and contextual preservation in video grounding?
• RQ4 Does the integration of SNNs and SSMs outperform conventional Transformer-based models in terms of accuracy and efficiency?
• RQ5 How do different spiking time steps and loss components affect the model’s performance on benchmark datasets?

Key findings

• SpikeMba achieves 85.6% Recall@0.5 and 75.5% mAP on the QVHighlights test set, outperforming state-of-the-art methods.
• The ablation study shows that removing the contextual moment reasoner (CMR) reduces mAP (Avg.) to 35.46% on the test set, highlighting its critical role in semantic reasoning.
• The spiking saliency detector improves performance by 4.8% in mAP (Avg.) compared to the baseline without SSD, demonstrating its effectiveness in saliency localization.
• Increasing spiking time steps from T=4 to T=8 boosts mAP (Avg.) from 42.03% to 44.81%, showing improved temporal processing with higher time-step resolution.
• The removal of contrastive loss reduces mAP (Avg.) to 41.70% on the test set, confirming its importance in learning discriminative feature representations.
• The model with relevant slots outperforms the variant without them, especially at higher time steps, proving their value in complex temporal reasoning.

Figure 2 : Architectural Overview of the Multi-Modal Spiking Saliency Mamba. The contextual moment reasoner dynamically leverages relevant slots for semantic association and inference. The spiking saliency detector generates a potential proposal set. The multi-modal relevant mamba block enhances lon