[Paper Review] DMDC: Dynamic-mask-based dual camera design for snapshot Hyperspectral Imaging

Zeyu Cai, Chengqian Jin+1 more|arXiv (Cornell University)|Aug 3, 2023

Photoacoustic and Ultrasonic Imaging4 citations

TL;DR

This paper proposes DMDC, a dynamic-mask-based dual-camera system combining an RGB camera and a CASSI camera to enhance snapshot hyperspectral imaging. By using RGB-derived spatial priors to dynamically optimize SLM masks and fusing multimodal data via cross-attention, the method achieves over 9 dB PSNR improvement over SOTA with significantly reduced parameters and FLOPs.

ABSTRACT

Deep learning methods are developing rapidly in coded aperture snapshot spectral imaging (CASSI). The number of parameters and FLOPs of existing state-of-the-art methods (SOTA) continues to increase, but the reconstruction accuracy improves slowly. Current methods still face two problems: 1) The performance of the spatial light modulator (SLM) is not fully developed due to the limitation of fixed Mask coding. 2) The single input limits the network performance. In this paper we present a dynamic-mask-based dual camera system, which consists of an RGB camera and a CASSI system running in parallel. First, the system learns the spatial feature distribution of the scene based on the RGB images, then instructs the SLM to encode each scene, and finally sends both RGB and CASSI images to the network for reconstruction. We further designed the DMDC-net, which consists of two separate networks, a small-scale CNN-based dynamic mask network for dynamic adjustment of the mask and a multimodal reconstruction network for reconstruction using RGB and CASSI measurements. Extensive experiments on multiple datasets show that our method achieves more than 9 dB improvement in PSNR over the SOTA. (https://github.com/caizeyu1992/DMDC)

Motivation & Objective

• Address the limited dynamic modulation capability of spatial light modulators (SLMs) in CASSI systems due to fixed mask coding.
• Overcome the performance bottleneck of single-input networks in hyperspectral reconstruction by leveraging complementary RGB and CASSI measurements.
• Improve reconstruction fidelity and efficiency by integrating spatial priors from RGB images to guide dynamic SLM encoding.
• Develop a multimodal, reversible-prior-based network architecture that fuses spectral and spatial features effectively without prior knowledge of camera response.
• Achieve state-of-the-art performance with lower computational cost (Params and FLOPs) than existing Transformer-based methods.

Proposed method

• Design a dual-camera system with parallel RGB and CASSI optical paths to capture complementary measurements in a snapshot.
• Implement a dynamic mask network that uses RGB image features to guide real-time SLM modulation, optimizing encoding for scene content.
• Propose DMDC-Net, a two-stream architecture with a small CNN-based dynamic mask network and a multimodal reconstruction network.
• Integrate cross-attention mechanisms between spectral and spatial features to exploit local similarity and global correlation in both domains.
• Embed a noise estimator and spatial attention modules to enhance robustness and feature learning in the reconstruction stream.
• Use a reversible prior framework to convert complex mapping into residual learning, enabling iterative refinement and improved optimization.

Experimental results

Research questions

• RQ1 Can dynamic mask adaptation guided by RGB priors significantly improve CASSI reconstruction quality compared to fixed or random masks?
• RQ2 To what extent does multimodal fusion of RGB and CASSI measurements enhance spatial and spectral fidelity in snapshot hyperspectral imaging?
• RQ3 How does the integration of cross-attention and reversible prior design improve reconstruction performance while reducing model complexity?
• RQ4 Can the proposed system achieve SOTA performance with substantially lower FLOPs and parameters than existing Transformer-based CASSI methods?
• RQ5 How does the dynamic mask strategy compare to manual or random mask coding in terms of reconstruction accuracy and robustness across diverse scenes?

Key findings

• DMDC-9stg achieves 49.14 dB PSNR, 99.49% SSIM, 0.0357 MRAE, and 0.0038 RMSE on the ARAD_1K dataset, significantly outperforming SOTA methods.
• The method improves PSNR by 15.85 dB over λ-net, 14.32 dB over TSA-net, and 10.65 dB over MST++ on ARAD_1K, with 7.86% higher SSIM than λ-net.
• DMDC-1stg achieves 4.62 dB higher PSNR than CST-L and 5.13 dB higher than MST++ while using only 35.0% of the parameters and 83.8% of the FLOPs.
• The dynamic mask component alone contributes as much to performance gain as the RGB branching, reducing MRAE by 0.0245 compared to fixed masks.
• The noise estimator improves PSNR by 0.80 dB, and cross-attention boosts performance by 1.00 dB on DMDC-1stg and 0.74 dB on DMDC-3stg.
• DMDC-5stg achieves over 49 dB PSNR at a frame rate exceeding 12 FPS, demonstrating high-speed reconstruction with superior accuracy.