[Paper Review] MFFW: A new dataset for multi-focus image fusion

Shuang Xu, Xiaoli Wei+3 more|arXiv (Cornell University)|Feb 12, 2020

Advanced Image Fusion Techniques37 references56 citations

TL;DR

This paper introduces MFFW, a real-world multi-focus image fusion dataset with defocus spread effect, consisting of 19 source image pairs (and ground-truth maps for 2-source cases), and evaluates 13 SOTA methods to show current methods struggle on DSE-rich scenes.

ABSTRACT

Multi-focus image fusion (MFF) is a fundamental task in the field of computational photography. Current methods have achieved significant performance improvement. It is found that current methods are evaluated on simulated image sets or Lytro dataset. Recently, a growing number of researchers pay attention to defocus spread effect, a phenomenon of real-world multi-focus images. Nonetheless, defocus spread effect is not obvious in simulated or Lytro datasets, where popular methods perform very similar. To compare their performance on images with defocus spread effect, this paper constructs a new dataset called MFF in the wild (MFFW). It contains 19 pairs of multi-focus images collected on the Internet. We register all pairs of source images, and provide focus maps and reference images for part of pairs. Compared with Lytro dataset, images in MFFW significantly suffer from defocus spread effect. In addition, the scenes of MFFW are more complex. The experiments demonstrate that most state-of-the-art methods on MFFW dataset cannot robustly generate satisfactory fusion images. MFFW can be a new baseline dataset to test whether an MMF algorithm is able to deal with defocus spread effect.

Motivation & Objective

• Motivate evaluation of multi-focus image fusion (MFF) under defocus spread effect (DSE) in real-world scenes.
• Provide a new dataset (MFFW) with challenging, real-world MFF pairs and annotated focus maps for robust benchmarking.
• Assess the performance of classic and deep learning MFF methods on DSE-rich data.
• Highlight limitations of current methods and motivate development of DSE-aware fusion approaches.

Proposed method

• Construct MFFW by collecting 19 multi-focus image pairs from the Internet, with 13 two-source and 6 multi-source sets.
• Register source image pairs and provide ground-truth focus maps for two-source pairs using manual annotation in LabelMe.
• Provide reference images for two-source pairs edited to align with human visual perception.
• Evaluate 13 SOTA methods (9 classic, 4 DL-based) on MFFW using no-reference fusion metrics.
• In DL experiments, train MMF-Net on NYU Depth V1 to synthesize defocus-based training data and generate focus maps.
• Employ 11 no-reference metrics (MI, TE, NCIE, GBM, SF, SSBM, CBM, LIF, AG, MSD, GLD) to assess fusion quality.

Experimental results

Research questions

• RQ1 Can current MFF methods robustly handle defocus spread effect in real-world scenes?
• RQ2 Do deep learning MFF models outperform classic transform/spatial-domain methods on the MFFW dataset?
• RQ3 How does performance vary between two-source and multi-source (3+) MFF tasks under DSE?
• RQ4 What are the limitations of ground-truth focus maps and reference images in evaluating MFF methods under DSE?

Key findings

• Most state-of-the-art methods fail to robustly generate satisfactory fusion images on MFFW.
• DL-based models do not show clear superiority over classic methods on MFFW datasets.
• MMF-Net can suffer from overfitting and may break down on some pairs.
• DL models trained on simulated data may not generalize well to real-world DSE-rich scenes.
• MFFW serves as a new benchmark to test whether MMF algorithms can deal with DSE.
• Fused results often exhibit artifacts or incorrect focus boundary handling due to DSE.