[Paper Review] From Text to Pixels: A Context-Aware Semantic Synergy Solution for Infrared and Visible Image Fusion

X. Rong Li, Yang Zou+5 more|arXiv (Cornell University)|Dec 31, 2023

Advanced Image Fusion Techniques4 citations

TL;DR

This paper proposes a text-guided, context-aware semantic synergy framework for infrared and visible image fusion that leverages CLIP-encoded textual descriptions to enhance feature alignment and object detection performance. By integrating a codebook for feature discretization and a bilevel optimization strategy to jointly optimize fusion and detection, the method achieves state-of-the-art mAP on multiple benchmarks, with a new paired IVIF+text dataset introduced as a key contribution.

ABSTRACT

With the rapid progression of deep learning technologies, multi-modality image fusion has become increasingly prevalent in object detection tasks. Despite its popularity, the inherent disparities in how different sources depict scene content make fusion a challenging problem. Current fusion methodologies identify shared characteristics between the two modalities and integrate them within this shared domain using either iterative optimization or deep learning architectures, which often neglect the intricate semantic relationships between modalities, resulting in a superficial understanding of inter-modal connections and, consequently, suboptimal fusion outcomes. To address this, we introduce a text-guided multi-modality image fusion method that leverages the high-level semantics from textual descriptions to integrate semantics from infrared and visible images. This method capitalizes on the complementary characteristics of diverse modalities, bolstering both the accuracy and robustness of object detection. The codebook is utilized to enhance a streamlined and concise depiction of the fused intra- and inter-domain dynamics, fine-tuned for optimal performance in detection tasks. We present a bilevel optimization strategy that establishes a nexus between the joint problem of fusion and detection, optimizing both processes concurrently. Furthermore, we introduce the first dataset of paired infrared and visible images accompanied by text prompts, paving the way for future research. Extensive experiments on several datasets demonstrate that our method not only produces visually superior fusion results but also achieves a higher detection mAP over existing methods, achieving state-of-the-art results.

Motivation & Objective

• To address the limitations of existing multi-modality image fusion methods that fail to model deep semantic relationships between infrared and visible images.
• To improve object detection accuracy by leveraging high-level semantics from textual descriptions to guide fusion.
• To establish a joint optimization framework that simultaneously enhances image fusion and downstream detection performance.
• To create the first paired dataset of infrared, visible, and corresponding text prompt images for multi-modality fusion research.

Proposed method

• Utilizes the CLIP model to encode textual prompts into high-level semantic embeddings that guide feature fusion between infrared and visible images.
• Employs a codebook to discretize continuous feature representations, improving training efficiency and generalization for detection tasks.
• Applies both self-attention and cross-attention mechanisms to enhance intra-modality and inter-modality feature learning.
• Implements a bilevel optimization strategy that jointly optimizes the fusion and detection objectives, enabling end-to-end performance improvement.
• Introduces two loss functions: content consistency loss to preserve input fidelity and structure loss to stabilize training.
• Designs a dual-branch network architecture that processes infrared and visible inputs separately before fusing them under textual guidance.

Experimental results

Research questions

• RQ1 Can textual descriptions significantly improve semantic alignment and fusion quality in infrared and visible image fusion?
• RQ2 How does joint optimization of fusion and detection via bilevel learning enhance downstream detection performance compared to sequential or independent training?
• RQ3 To what extent do cross-attention and self-attention mechanisms contribute to feature refinement and modality interaction in the fusion process?
• RQ4 How effective is CLIP-based text guidance in focusing the fusion network on salient, semantically relevant features?
• RQ5 What is the impact of different textual prompt types (coarse vs. fine) on detection accuracy and fusion quality?

Key findings

• The proposed method achieves state-of-the-art mAP of 0.517 on the M3FD dataset, outperforming prior methods.
• The ablation study shows that using both self-attention and cross-attention improves mAP by 0.099 compared to using only one mechanism.
• The inclusion of both content consistency loss and structure loss leads to a 0.090 mAP improvement over training without either.
• The model with fine-tuned text prompts detects previously unmarked objects more accurately than ground-truth annotations in some cases.
• Visual results demonstrate that textual prompts enhance brightness and highlight key features, improving perceptual fusion quality.
• The new paired IVIF+text dataset enables more robust evaluation and future research in text-guided multi-modality fusion.