[Paper Review] Spatio-channel Attention Blocks for Cross-modal Crowd Counting

Youjia Zhang, So-Yun Choi+1 more|arXiv (Cornell University)|Oct 19, 2022

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes Cross-modal Spatio-Channel Attention (CSCA) blocks, a plug-and-play module that enhances cross-modal crowd counting by jointly modeling spatial and channel-wise attention across RGB, thermal, and depth modalities. The method improves performance across diverse backbones on RGB-T and RGB-D datasets, achieving state-of-the-art results by effectively fusing multi-modal features with low computational overhead.

ABSTRACT

Crowd counting research has made significant advancements in real-world applications, but it remains a formidable challenge in cross-modal settings. Most existing methods rely solely on the optical features of RGB images, ignoring the feasibility of other modalities such as thermal and depth images. The inherently significant differences between the different modalities and the diversity of design choices for model architectures make cross-modal crowd counting more challenging. In this paper, we propose Cross-modal Spatio-Channel Attention (CSCA) blocks, which can be easily integrated into any modality-specific architecture. The CSCA blocks first spatially capture global functional correlations among multi-modality with less overhead through spatial-wise cross-modal attention. Cross-modal features with spatial attention are subsequently refined through adaptive channel-wise feature aggregation. In our experiments, the proposed block consistently shows significant performance improvement across various backbone networks, resulting in state-of-the-art results in RGB-T and RGB-D crowd counting.

Motivation & Objective

• To address the challenge of fusing heterogeneous features from multiple modalities—such as RGB, thermal, and depth—in crowd counting under varying lighting and occlusion conditions.
• To develop a lightweight, reusable module that can be seamlessly integrated into any existing RGB-based crowd counting backbone without architectural redesign.
• To overcome limitations of naive fusion strategies (e.g., early or late fusion) that degrade performance due to modality misalignment and noise.
• To enable adaptive, dynamic feature aggregation that captures both inter-modal and intra-modal correlations across spatial and channel dimensions.

Proposed method

• The CSCA block employs a Spatial-wise Cross-modal Attention (SCA) module that computes cross-modal attention using query, key, and value features across modalities, enabling global spatial correlation modeling with reduced computation.
• SCA reassembles feature maps at the spatial level to efficiently capture long-range dependencies between different modalities, avoiding the high cost of standard non-local blocks.
• The Channel-wise Feature Aggregation (CFA) module adaptively recalibrates the spatially attended features by learning channel-wise weights, enhancing discriminative representation.
• The CSCA block is designed as a plug-and-play module, allowing integration into any pre-trained unimodal backbone (e.g., BL, ResNet) without retraining from scratch.
• The method uses a learnable attention mechanism that dynamically adjusts attention weights based on modality-specific feature distributions, improving robustness to noise and illumination changes.
• The architecture is evaluated on RGB-T and RGB-D datasets, demonstrating consistent gains across multiple backbone networks and fusion strategies.

Experimental results

Research questions

• RQ1 Can a lightweight, modular attention mechanism effectively fuse multi-modal features (RGB, thermal, depth) for crowd counting without architectural overhaul?
• RQ2 How does the proposed CSCA block compare to standard non-local or naive fusion methods in terms of accuracy and computational efficiency?
• RQ3 To what extent can CSCA improve performance across different illumination conditions and noise levels in cross-modal settings?
• RQ4 Does the complementary fusion of multi-modal data consistently outperform unimodal (RGB-only or single-modality) approaches in challenging crowd counting scenarios?

Key findings

• The CSCA block achieves state-of-the-art performance on both RGBT-CC and ShanghaiTechRGBD datasets, outperforming existing methods across multiple backbone networks.
• On the ShanghaiTechRGBD dataset, the proposed method achieves a RMSE of 24.74 with the BL backbone, significantly outperforming unimodal RGB (RMSE: 87.29) and thermal-only (RMSE: 28.45) baselines under low-light conditions.
• Ablation studies confirm that removing either the SCA or CFA component leads to a severe drop in performance, demonstrating the complementary role of both modules.
• Replacing SCA with a standard non-local block causes out-of-memory errors on high-resolution images (1920×1080), highlighting CSCA’s superior computational efficiency.
• Visualization results show that CSCA effectively leverages thermal data in low-light scenes and reduces noise-induced errors in depth data, improving localization accuracy.
• The method consistently improves performance across different input settings, including unimodal RGB, unimodal thermal, and multimodal RGB-T, confirming its robustness and generalization capability.