[Paper Review] BAANet: Learning Bi-directional Adaptive Attention Gates for Multispectral Pedestrian Detection

Xiaoxiao Yang, Yeqian Qiang+3 more|arXiv (Cornell University)|Dec 4, 2021

Video Surveillance and Tracking Methods4 citations

TL;DR

This paper proposes BAANet, a novel multispectral pedestrian detection framework that uses Bi-directional Adaptive Attention Gates (BAA-Gate) to progressively distill modality-specific noise and recalibrate cross-modality features through a multi-stage fusion strategy. The method achieves state-of-the-art performance on the KAIST dataset with a 6.98% miss rate in night conditions and 7.92% overall log-average miss rate, while maintaining high inference speed at 0.07s per image.

ABSTRACT

Thermal infrared (TIR) image has proven effectiveness in providing temperature cues to the RGB features for multispectral pedestrian detection. Most existing methods directly inject the TIR modality into the RGB-based framework or simply ensemble the results of two modalities. This, however, could lead to inferior detection performance, as the RGB and TIR features generally have modality-specific noise, which might worsen the features along with the propagation of the network. Therefore, this work proposes an effective and efficient cross-modality fusion module called Bi-directional Adaptive Attention Gate (BAA-Gate). Based on the attention mechanism, the BAA-Gate is devised to distill the informative features and recalibrate the representations asymptotically. Concretely, a bi-direction multi-stage fusion strategy is adopted to progressively optimize features of two modalities and retain their specificity during the propagation. Moreover, an adaptive interaction of BAA-Gate is introduced by the illumination-based weighting strategy to adaptively adjust the recalibrating and aggregating strength in the BAA-Gate and enhance the robustness towards illumination changes. Considerable experiments on the challenging KAIST dataset demonstrate the superior performance of our method with satisfactory speed.

Motivation & Objective

• To address the challenge of modality-specific noise in RGB and thermal infrared (TIR) images, which degrades feature quality during deep network propagation.
• To improve cross-modality feature fusion by explicitly modeling inter-modality correlations and suppressing noise in a bi-directional, multi-stage manner.
• To enhance robustness under varying illumination conditions by adaptively adjusting fusion strength using an illumination-based weighting strategy.
• To achieve high detection accuracy and efficiency simultaneously, balancing performance and inference speed in multispectral pedestrian detection.

Proposed method

• Proposes a Bi-directional Adaptive Attention Gate (BAA-Gate) that first suppresses noisy features in one modality (e.g., RGB in low light) and then uses the refined features to recalibrate the other (e.g., TIR).
• Employs a multi-stage fusion strategy where features are progressively refined and recalibrated across multiple network stages to preserve modality specificity and enhance discriminability.
• Introduces an illumination-based weighting strategy to adaptively control the strength of recalibration and aggregation in BAA-Gate, improving robustness to lighting variations.
• Utilizes deformable convolutional networks (DCN) for modality alignment to improve feature correspondence between RGB and TIR branches.
• Employs a two-branch CNN architecture with late fusion via BAA-Gate, replacing simple concatenation or early fusion to improve representation quality.
• Trains end-to-end with a standard detection head (e.g., Faster R-CNN) using cross-entropy and box regression losses.

Experimental results

Research questions

• RQ1 Can a bi-directional, multi-stage attention mechanism effectively suppress modality-specific noise in RGB and TIR features during deep network inference?
• RQ2 How does adaptive recalibration of features through BAA-Gate improve detection accuracy under challenging lighting and occlusion conditions?
• RQ3 To what extent does the illumination-based weighting strategy enhance robustness to varying environmental lighting in multispectral pedestrian detection?
• RQ4 Does the proposed BAA-Gate enable better feature alignment and representation learning compared to simple concatenation or early fusion strategies?

Key findings

• BAANet achieves the lowest overall log-average miss rate of 7.92% on the KAIST dataset, outperforming all other state-of-the-art methods.
• In the night subset, BAANet achieves a 6.98% miss rate, surpassing the second-best method by 0.88 percentage points.
• On the far-scale subset, BAANet achieves a 51.25% miss rate, outperforming the second-best method (CIAN) by 4.57 percentage points.
• The ablation study confirms that all components—BAA-Gate, illumination-based weighting, and modality alignment—contribute significantly to performance, with the full model achieving the best result.
• BAANet maintains high inference speed at 0.07 seconds per image, comparable to the fastest existing methods, demonstrating a strong accuracy-speed trade-off.
• Qualitative results show BAANet generalizes well to challenging scenarios, including low-light, small-scale, and occluded pedestrians, with fewer false and missed detections than competing models.