[Paper Review] TGFuse: An Infrared and Visible Image Fusion Approach Based on Transformer and Generative Adversarial Network

Dongyu Rao, Xiao‐Jun Wu+1 more|arXiv (Cornell University)|Jan 25, 2022

Advanced Image Fusion Techniques18 citations

TL;DR

This paper proposes TGFuse, an infrared and visible image fusion method that combines a lightweight transformer module for global feature modeling with generative adversarial networks (GANs) to enhance fusion quality. By leveraging cross-channel and spatial attention in a transformer-based fusion module and using dual discriminators to preserve modality-specific details, TGFuse achieves state-of-the-art performance in both subjective and objective evaluations, outperforming existing methods on multiple benchmark metrics.

ABSTRACT

The end-to-end image fusion framework has achieved promising performance, with dedicated convolutional networks aggregating the multi-modal local appearance. However, long-range dependencies are directly neglected in existing CNN fusion approaches, impeding balancing the entire image-level perception for complex scenario fusion. In this paper, therefore, we propose an infrared and visible image fusion algorithm based on a lightweight transformer module and adversarial learning. Inspired by the global interaction power, we use the transformer technique to learn the effective global fusion relations. In particular, shallow features extracted by CNN are interacted in the proposed transformer fusion module to refine the fusion relationship within the spatial scope and across channels simultaneously. Besides, adversarial learning is designed in the training process to improve the output discrimination via imposing competitive consistency from the inputs, reflecting the specific characteristics in infrared and visible images. The experimental performance demonstrates the effectiveness of the proposed modules, with superior improvement against the state-of-the-art, generalising a novel paradigm via transformer and adversarial learning in the fusion task.

Motivation & Objective

• To address the limitation of CNN-based fusion methods in capturing long-range spatial and channel-wise dependencies in infrared and visible image fusion.
• To improve fusion quality by integrating global context modeling through a lightweight transformer module tailored for multi-modal image fusion.
• To enhance the discriminative quality of fused images by incorporating adversarial learning that preserves modality-specific characteristics.
• To establish a novel end-to-end fusion framework that combines the strengths of transformers and GANs for unsupervised image fusion tasks.
• To validate the effectiveness of the proposed architecture through comprehensive ablation studies and benchmark comparisons.

Proposed method

• A channel-token transformer is introduced to model inter-channel relationships in feature maps, enabling joint spatial and channel-wise fusion learning.
• A spatial transformer module is designed to capture long-range spatial dependencies across the image, improving global fusion representation.
• The composite transformer fusion module combines spatial and channel-wise attention to refine feature interactions across modality-specific features.
• Adversarial training is implemented using two discriminators—one comparing infrared and fused images, and another comparing visible and fused images—to enforce fidelity to input modality characteristics.
• A lightweight encoder-decoder network with residual blocks is used for initial feature extraction, followed by the transformer fusion module.
• Position embeddings are omitted in the transformer to reduce inductive bias, as they were found to degrade performance in ablation studies.

Experimental results

Research questions

• RQ1 Can a lightweight transformer module effectively model long-range spatial and channel-wise dependencies in infrared and visible image fusion?
• RQ2 How does adversarial learning improve the perceptual quality and modality fidelity of fused images in an unsupervised setting?
• RQ3 What is the optimal configuration of transformer layers, CNN depth, and feature channel dimensions for image fusion performance?
• RQ4 Does removing position embeddings in the transformer improve or harm fusion performance in this task?
• RQ5 How do the individual components (spatial, channel, and adversarial modules) contribute to the overall fusion quality compared to ablated variants?

Key findings

• The proposed TGFuse method achieves superior performance across all objective evaluation metrics compared to state-of-the-art methods, with consistent improvements on multiple benchmarks.
• The ablation study shows that using both spatial and channel transformers in sequence (channel first, then spatial) yields the best fusion results, outperforming individual or reversed configurations.
• Removing position embeddings from the transformer module leads to improved performance, indicating that positional bias is not beneficial for this fusion task.
• The optimal number of transformer encoder layers is four; fewer layers (three) fail to learn meaningful fusion relationships, while deeper networks (five) introduce redundancy and degrade results.
• A CNN depth of four residual blocks yields the best feature representation; deeper networks result in meaningless black outputs due to excessive downsampling.
• The optimal number of feature channels in the fusion network is 64, as determined by ablation, with other values leading to suboptimal performance on evaluation metrics.