[Paper Review] End-to-End Learning for Simultaneously Generating Decision Map and Multi-Focus Image Fusion Result

Boyuan Ma, Xiang Yin+2 more|arXiv (Cornell University)|Oct 17, 2020

Advanced Image Fusion TechniquesEngineering58 references78 citations

TL;DR

This paper proposes GACN, a cascade network that end-to-end simultaneously generates a decision map and fused multi-focus image, avoiding empirical post-processing. By introducing a gradient-aware loss and decision calibration strategy, it improves fusion quality and accelerates multiple-image fusion by over 30%.

ABSTRACT

The general aim of multi-focus image fusion is to gather focused regions of different images to generate a unique all-in-focus fused image. Deep learning based methods become the mainstream of image fusion by virtue of its powerful feature representation ability. However, most of the existing deep learning structures failed to balance fusion quality and end-to-end implementation convenience. End-to-end decoder design often leads to unrealistic result because of its non-linear mapping mechanism. On the other hand, generating an intermediate decision map achieves better quality for the fused image, but relies on the rectification with empirical post-processing parameter choices. In this work, to handle the requirements of both output image quality and comprehensive simplicity of structure implementation, we propose a cascade network to simultaneously generate decision map and fused result with an end-to-end training procedure. It avoids the dependence on empirical post-processing methods in the inference stage. To improve the fusion quality, we introduce a gradient aware loss function to preserve gradient information in output fused image. In addition, we design a decision calibration strategy to decrease the time consumption in the application of multiple images fusion. Extensive experiments are conducted to compare with 19 different state-of-the-art multi-focus image fusion structures with 6 assessment metrics. The results prove that our designed structure can generally ameliorate the output fused image quality, while implementation efficiency increases over 30\% for multiple images fusion.

Motivation & Objective

• To address the trade-off between fusion quality and end-to-end training simplicity in deep learning-based image fusion.
• To eliminate reliance on empirical post-processing for decision map rectification in fused image generation.
• To preserve gradient information in fused images through a novel loss function.
• To enhance computational efficiency in multi-image fusion using a decision calibration strategy.

Proposed method

• Proposes a cascade network architecture that jointly predicts decision maps and fused images in an end-to-end manner.
• Introduces a gradient-aware loss function (LQg) to preserve edge and texture details by optimizing gradient similarity in the fused output.
• Employs a decision calibration module that computes activity levels across multiple input images to reduce redundant computation in multi-image fusion.
• Uses a combination of LDice and LQg loss functions to improve decision map accuracy and gradient preservation.
• Applies a smooth approximation and absolute function in the gradient loss for numerical stability and performance.
• Trains the network end-to-end without post-processing, ensuring direct optimization of fused image quality.

Experimental results

Research questions

• RQ1 Can end-to-end training simultaneously generate high-quality decision maps and fused images without post-processing?
• RQ2 How does a gradient-aware loss function improve the perceptual quality of fused multi-focus images?
• RQ3 Can a decision calibration strategy significantly reduce time consumption in multi-image fusion?
• RQ4 How does the proposed method compare to 19 state-of-the-art methods in fusion quality and efficiency?

Key findings

• The proposed GACN achieves superior fusion quality across six evaluation metrics, outperforming 19 SOTA methods.
• The combination of LDice and LQg loss functions yields the best performance, with LQg being most effective in preserving gradient information.
• The decision calibration strategy reduces time consumption by 30.65% for the 'chip' dataset compared to serial fusion, with a 30%+ efficiency gain across methods.
• The ablation study confirms that LQg is more effective than LEG and LST in preserving structural details.
• The method achieves robust performance on diverse datasets, including microscopic images with multiple focus points.
• The visualization shows that GACN produces clearer fused images with fewer artifacts than CNN-Fuse and SESF-Fuse.