[Paper Review] Illumination-aware Faster R-CNN for Robust Multispectral Pedestrian Detection

Chengyang Li, Dan Song+2 more|arXiv (Cornell University)|Mar 14, 2018

Remote-Sensing Image Classification44 references20 citations

TL;DR

This paper proposes Illumination-aware Faster R-CNN (IAF R-CNN), a multispectral pedestrian detection framework that adaptively fuses color and thermal features using an illumination-aware gating mechanism. By estimating illumination conditions via an IAN-based network and dynamically weighting modality contributions, IAF R-CNN achieves state-of-the-art performance on the KAIST multispectral benchmark, outperforming existing methods by up to 5.03% in MR^I under challenging lighting conditions.

ABSTRACT

Multispectral images of color-thermal pairs have shown more effective than a single color channel for pedestrian detection, especially under challenging illumination conditions. However, there is still a lack of studies on how to fuse the two modalities effectively. In this paper, we deeply compare six different convolutional network fusion architectures and analyse their adaptations, enabling a vanilla architecture to obtain detection performances comparable to the state-of-the-art results. Further, we discover that pedestrian detection confidences from color or thermal images are correlated with illumination conditions. With this in mind, we propose an Illumination-aware Faster R-CNN (IAF RCNN). Specifically, an Illumination-aware Network is introduced to give an illumination measure of the input image. Then we adaptively merge color and thermal sub-networks via a gate function defined over the illumination value. The experimental results on KAIST Multispectral Pedestrian Benchmark validate the effectiveness of the proposed IAF R-CNN.

Motivation & Objective

• To address the challenge of robust pedestrian detection under varying illumination conditions, especially in low-light or nighttime scenarios.
• To investigate effective fusion strategies for multispectral (color-thermal) features in Faster R-CNN-based detectors.
• To develop a dynamic fusion mechanism that adapts to illumination levels, improving detection reliability across lighting conditions.
• To evaluate and compare multiple convolutional network fusion architectures for multispectral pedestrian detection.
• To demonstrate that illumination-aware fusion significantly enhances detection performance compared to fixed or heuristic fusion strategies.

Proposed method

• An illumination-aware network (IAN) is trained to estimate the illumination level of input images using only day/night binary labels.
• A gate function computes modality-specific weights based on the illumination measure, enabling dynamic fusion of color and thermal sub-network outputs.
• The color and thermal sub-networks are jointly trained with the illumination-aware fusion module within a unified Faster R-CNN framework.
• The fusion mechanism adaptively increases thermal network weight under poor illumination, while favoring color features in well-lit conditions.
• Three illumination estimation methods (Key, Range, IAN) are evaluated, with IAN showing superior generalization on complex outdoor scenes.
• The model is end-to-end trainable, allowing joint optimization of feature extraction, illumination estimation, and fusion gating.

Experimental results

Research questions

• RQ1 How do different multispectral fusion architectures derived from Faster R-CNN compare in performance and adaptability for pedestrian detection?
• RQ2 To what extent does illumination condition affect the reliability of color versus thermal modality in pedestrian detection?
• RQ3 Can an illumination-aware fusion mechanism improve detection robustness across varying lighting conditions?
• RQ4 How does the proposed illumination-aware weighting compare to fixed-weighting (e.g., average) or threshold-based (hard 0-1) fusion strategies?
• RQ5 Can a lightweight illumination estimator trained on weak labels (day/night) effectively guide modality fusion in real-world multispectral detection?

Key findings

• The proposed IAF R-CNN achieves state-of-the-art performance on the KAIST multispectral pedestrian detection benchmark, outperforming prior methods.
• The illumination-aware weighting mechanism improves detection performance by 0.67% over average weighting and 5.03% over hard 0-1 weighting in terms of MR^I.
• Using IAN for illumination estimation yields significantly better results than Key or Range methods, especially in complex outdoor scenes.
• The model demonstrates robustness in both daytime and nighttime conditions, with dynamic fusion favoring thermal modality in low-light scenarios.
• A vanilla multispectral Faster R-CNN, once properly adapted, achieves performance close to state-of-the-art, indicating high potential in baseline architectures.
• Visualization confirms that the learned gate function effectively adjusts modality weights based on estimated illumination, aligning with intuitive expectations.