[Paper Review] Beyond Night Visibility: Adaptive Multi-Scale Fusion of Infrared and Visible Images

Shufan Pei, Junhong Lin+3 more|arXiv (Cornell University)|Mar 2, 2024

Advanced Image Fusion Techniques4 citations

TL;DR

This paper proposes AMFusion, an adaptive multi-scale fusion network that enhances nighttime visibility by jointly addressing low-light and light-effect degradation using infrared and visible images. By separately fusing spatial (illumination-aware) and semantic (detection-guided) features with a novel illumination loss, AMFusion achieves superior visual quality and 61.7% mAP50→95 on pedestrian detection, outperforming state-of-the-art methods.

ABSTRACT

In addition to low light, night images suffer degradation from light effects (e.g., glare, floodlight, etc). However, existing nighttime visibility enhancement methods generally focus on low-light regions, which neglects, or even amplifies the light effects. To address this issue, we propose an Adaptive Multi-scale Fusion network (AMFusion) with infrared and visible images, which designs fusion rules according to different illumination regions. First, we separately fuse spatial and semantic features from infrared and visible images, where the former are used for the adjustment of light distribution and the latter are used for the improvement of detection accuracy. Thereby, we obtain an image free of low light and light effects, which improves the performance of nighttime object detection. Second, we utilize detection features extracted by a pre-trained backbone that guide the fusion of semantic features. Hereby, we design a Detection-guided Semantic Fusion Module (DSFM) to bridge the domain gap between detection and semantic features. Third, we propose a new illumination loss to constrain fusion image with normal light intensity. Experimental results demonstrate the superiority of AMFusion with better visual quality and detection accuracy. The source code will be released after the peer review process.

Motivation & Objective

• To address the limitations of existing nighttime visibility enhancement methods that amplify light effects while improving low-light regions.
• To develop a multi-modality fusion framework that simultaneously mitigates low-light and overexposure issues in nighttime images.
• To improve object detection accuracy in low-light conditions by leveraging detection-guided feature fusion.
• To design an unsupervised illumination loss that constrains fusion outputs to maintain normal light intensity.
• To bridge the domain gap between detection and semantic features through a detection-guided fusion module.

Proposed method

• AMFusion employs separate feature extraction for spatial and semantic representations from infrared and visible images.
• An Illumination-aware Detection Fusion Module (IDFM) fuses spatial features using illumination information to correct light distribution.
• A Detection-guided Semantic Fusion Module (DSFM) uses pre-trained detection features to guide semantic feature fusion, improving detection accuracy.
• Spatial and fused semantic features are combined to generate a final fusion image with enhanced visual quality.
• A novel unsupervised illumination loss, composed of intensity and exposure control components, ensures realistic light distribution in the output.
• The model is trained end-to-end with a combination of perceptual, detection, and illumination losses to optimize for both visual fidelity and detection performance.

Experimental results

Research questions

• RQ1 Can multi-scale fusion of infrared and visible images effectively suppress both low-light and light-effect degradations in nighttime images?
• RQ2 How can detection features from a pre-trained model improve semantic feature fusion in infrared-visible image fusion?
• RQ3 To what extent does an unsupervised illumination loss improve the realism and quality of fused nighttime images?
• RQ4 Does adaptive fusion based on illumination regions lead to better detection performance compared to uniform fusion strategies?
• RQ5 What is the relative contribution of spatial, semantic, and illumination-aware modules to overall fusion performance?

Key findings

• AMFusion achieves the highest mAP50→95 of 61.7% on the M³FD+LLVIP dataset, outperforming all baseline methods.
• The model with all components (IDFM, DSFM, SRM, and full illumination loss) achieves the best detection performance, with a 7.323 mAP50→95 score.
• Abllation studies confirm that each module—IDFM, DSFM, and SRM—contributes incrementally to improved detection accuracy and visual quality.
• The illumination loss significantly improves light distribution, with the full loss setting reducing overexposure and enhancing realism, as shown in visual comparisons.
• The detection-guided semantic fusion module (DSFM) enables better feature alignment and target focus, leading to accurate detection of small objects like pedestrians.
• Removal of the illumination loss leads to a drop in mAP50→95 from 61.7% to 60.4%, demonstrating its critical role in maintaining realistic lighting.