[Paper Review] Deep demosaicking for multispectral filter arrays

Kazuma Shinoda, Shoichiro Yoshiba+1 more|arXiv (Cornell University)|Aug 24, 2018

Advanced Image Fusion Techniques9 references20 citations

TL;DR

This paper proposes a deep learning-based demosaicking method for multispectral images using a two-stage approach: bilinear interpolation followed by residual network (ResNet) refinement with 3D convolutional layers. The method achieves a 43.05 dB average PSNR on the CAVE dataset, significantly outperforming bilinear interpolation (34.58 dB) and PPID (40.38 dB), demonstrating superior artifact suppression and image quality.

ABSTRACT

We propose a novel demosaicking method for multispectral filter arrays based on a deep convolutional neural network. The proposed method first interpolates mosaicked multispectral images utilizing a bilinear approach, then applies a residual network to initial demosaicked images. The residual network consists of various three-dimensional convolutional layers and a rectified linear unit for describing the features of a multispectral data cube. Experimental results reveal that the proposed method outperforms conventional demosaicking methods.

Motivation & Objective

• Address the challenge of reconstructing full-resolution multispectral images from mosaicked data captured via multispectral filter arrays (MSFAs), which suffer from high missing pixel rates compared to RGB Bayer patterns.
• Improve demosaicking quality by leveraging deep learning to reduce color and edge artifacts common in conventional interpolation methods.
• Develop a novel two-stage framework combining bilinear interpolation with a 3D-convolutional residual network to enhance feature representation across spatial and spectral dimensions.
• Demonstrate that residual learning and 3D convolutions are effective for multispectral demosaicking, especially in preserving fine details and reducing reconstruction errors.

Proposed method

• First, apply bilinear interpolation using an N×N window (N = MSFA block size + 1) to each spectral band independently to generate an initial demosaicked image.
• Then, refine the initial image using a deep residual network (ResNet) with six modules, each containing 3D convolutional layers and ReLU activation functions.
• Each ResNet module uses shortcut connections with 1×1×1 convolutions to match feature dimensions, enabling residual learning and mitigating degradation.
• The final layer combines 32 feature maps into a single residual map, which is added to the initial demosaicked image to produce the final output.
• Employ 3D convolutions (e.g., 3×3×3 or 1×1×1 kernels) that process spatial and spectral dimensions simultaneously, capturing local signal variations in multispectral data cubes.
• Train the network using 8-fold cross-validation on 32 images from the CAVE dataset, with batch size 8 to manage memory and computation.

Experimental results

Research questions

• RQ1 Can a two-stage deep learning framework combining bilinear interpolation and 3D-ResNet improve multispectral image demosaicking over conventional methods?
• RQ2 To what extent do residual connections and 3D convolutions reduce artifacts and improve PSNR in demosaicked multispectral images?
• RQ3 How does the proposed method compare to bilinear interpolation and the PPI difference (PPID) method in terms of PSNR and visual quality?
• RQ4 What is the contribution of 3D convolution versus 2D convolution in preserving spectral-spatial features during demosaicking?
• RQ5 How sensitive is the performance to the removal of shortcut connections in the residual network?

Key findings

• The proposed method achieved an average PSNR of 43.05 dB on the CAVE dataset, outperforming bilinear interpolation (34.58 dB) and PPID (40.38 dB) by 8.47 dB and 2.67 dB, respectively.
• The method reduced blurring and edge artifacts: letters in 'fake and real peppers' were clearly readable in the proposed output, unlike in bilinear results.
• Edge artifacts present in PPID results were suppressed in the proposed method, as confirmed in 'clay' and 'beads' images.
• Color artifacts persisted in 'fake and real tomatoes' due to residual errors from the initial bilinear step, indicating a need for improved first-stage preprocessing.
• Removing all shortcut connections reduced average PSNR by 5.05 dB, confirming the critical role of residual learning in performance.
• Replacing 3D convolutions with 2D convolutions decreased PSNR by 1.19 dB, proving that 3D convolutions better capture joint spatial-spectral features.