[Paper Review] MFIF-GAN: A New Generative Adversarial Network for Multi-Focus Image Fusion

Yicheng Wang, Shuang Xu+4 more|arXiv (Cornell University)|Sep 21, 2020

Advanced Image Fusion Techniques52 references4 citations

TL;DR

This paper proposes MFIF-GAN, a novel generative adversarial network for multi-focus image fusion that reduces the defocus spread effect (DSE) by generating focus maps where foreground regions are slightly larger than actual objects. The method uses a six-branch encoder with Squeeze-and-Excitation Residual blocks, combines L1 reconstruction and gradient penalty in the loss function, and is trained on a synthetic α-matte dataset, achieving state-of-the-art performance in visual quality, quantitative metrics, and inference speed.

ABSTRACT

Multi-Focus Image Fusion (MFIF) is a promising image enhancement technique to obtain all-in-focus images meeting visual needs and it is a precondition of other computer vision tasks. One of the research trends of MFIF is to avoid the defocus spread effect (DSE) around the focus/defocus boundary (FDB). In this paper,we propose a network termed MFIF-GAN to attenuate the DSE by generating focus maps in which the foreground region are correctly larger than the corresponding objects. The Squeeze and Excitation Residual module is employed in the network. By combining the prior knowledge of training condition, this network is trained on a synthetic dataset based on an α-matte model. In addition, the reconstruction and gradient regularization terms are combined in the loss functions to enhance the boundary details and improve the quality of fused images. Extensive experiments demonstrate that the MFIF-GAN outperforms several state-of-the-art (SOTA) methods in visual perception, quantitative analysis as well as efficiency. Moreover, the edge diffusion and contraction module is firstly proposed to verify that focus maps generated by our method are accurate at the pixel level.

Motivation & Objective

• To address the defocus spread effect (DSE) in multi-focus image fusion, which degrades image quality near focus/defocus boundaries.
• To improve fusion quality by generating focus maps where foreground regions are slightly larger than actual objects, simulating real DSE.
• To develop a deep learning-based fusion framework that outperforms existing state-of-the-art methods in visual perception, quantitative metrics, and computational efficiency.
• To create a realistic synthetic dataset with DSE for training supervised MFIF models.
• To validate the accuracy of generated focus maps through a novel edge diffusion and contraction module at the pixel level.

Proposed method

• A six-branch encoder with Squeeze-and-Excitation Residual (SE-ResNet) blocks is used to extract multi-scale features from input images.
• The generator is trained on a synthetic dataset generated using an α-matte model applied to the PASCAL VOC 2012 dataset, simulating realistic DSE.
• A combined loss function includes L1-norm reconstruction loss and gradient penalty (L_gp) to enhance boundary detail preservation.
• Adversarial training with a discriminator is employed to improve realism and structural quality of fused images.
• A lightweight post-processing step using a SRR (super-resolution reconstruction) algorithm refines initial focus maps efficiently.
• An edge diffusion and contraction module is introduced to validate pixel-level accuracy of focus map generation.

Experimental results

Research questions

• RQ1 Can a deep generative model effectively reduce the defocus spread effect (DSE) in multi-focus image fusion by generating more realistic focus maps?
• RQ2 How does the use of an α-matte-based synthetic dataset impact the performance of a supervised MFIF model?
• RQ3 To what extent do L1 reconstruction and gradient penalty losses improve edge detail preservation in fused images?
• RQ4 Does the proposed six-branch, attention-based network architecture outperform standard CNNs in fusion quality and robustness?
• RQ5 Can a post-processing strategy refine focus maps efficiently without compromising speed or quality?

Key findings

• MFIF-GAN achieves the best performance across all quantitative metrics (e.g., PSNR, SSIM, LIF, AG, MSD, GLD) compared to state-of-the-art methods on multiple benchmarks.
• The method reduces inference time to 0.0486s on RGB images, 0.047s on MFFW, and 0.0133s on grayscale, outperforming SOTA in efficiency.
• Ablation studies confirm that the L1 reconstruction loss and gradient penalty significantly improve edge detail quality, with BCE-based loss leading to worse performance on edge metrics.
• The use of the α-matte dataset and SE-ResNet blocks leads to superior feature extraction, especially in handling DSE-affected regions.
• The post-processing step enhances focus map accuracy with minimal computational cost, maintaining the fastest overall inference speed.
• The edge diffusion and contraction module confirms that generated focus maps are pixel-accurate, validating the method's ability to simulate and mitigate DSE at the object boundary level.