[Paper Review] Mixture-Net: Low-Rank Deep Image Prior Inspired by Mixture Models for Spectral Image Recovery

Tatiana Gelvez, Jorge Bacca+1 more|arXiv (Cornell University)|Nov 5, 2022

Image and Signal Denoising Methods4 citations

TL;DR

Mixture-Net is a non-data-driven deep image prior network for spectral image recovery that integrates interpretable linear and non-linear low-rank mixture models into a deep generative architecture. By modeling the image as a weighted sum of endmembers and abundances, it achieves state-of-the-art performance in denoising, super-resolution, and compressive sensing without requiring training data, while enabling direct unmixing and material identification via learned interpretable features.

ABSTRACT

This paper proposes a non-data-driven deep neural network for spectral image recovery problems such as denoising, single hyperspectral image super-resolution, and compressive spectral imaging reconstruction. Unlike previous methods, the proposed approach, dubbed Mixture-Net, implicitly learns the prior information through the network. Mixture-Net consists of a deep generative model whose layers are inspired by the linear and non-linear low-rank mixture models, where the recovered image is composed of a weighted sum between the linear and non-linear decomposition. Mixture-Net also provides a low-rank decomposition interpreted as the spectral image abundances and endmembers, helpful in achieving remote sensing tasks without running additional routines. The experiments show the MixtureNet effectiveness outperforming state-of-the-art methods in recovery quality with the advantage of architecture interpretability.

Motivation & Objective

• To address the limitations of black-box deep learning in spectral image recovery by introducing interpretability through physical priors.
• To develop a non-data-driven method that implicitly learns image priors without requiring large labeled datasets.
• To unify spectral image recovery with downstream remote sensing tasks like unmixing and material identification through interpretable feature learning.
• To improve recovery quality in ill-posed problems such as denoising, super-resolution, and compressive sensing using model-based regularizers.
• To enable end-to-end recovery and analysis by embedding abundance and endmember estimation directly into the network architecture.

Proposed method

• Mixture-Net employs a deep generative architecture composed of multiple interpretable deep-blocks, each modeling a linear and non-linear mixture of endmembers and abundances.
• The input is pre-processed via Tucker decomposition to enforce low-rank structure, reflecting the intrinsic low-dimensionality of spectral images.
• Each deep-block consists of three trainable layers: one for estimating abundance maps, one for learning endmember spectra, and one for modeling non-linear spectral interactions.
• The loss function combines reconstruction loss with custom regularizers that enforce spatial-spectral correlations and physical plausibility of the learned components.
• The network is trained end-to-end using a deep image prior paradigm, initializing with random weights and optimizing for a single image without external data.
• Interpretability is achieved by directly associating learned features and weights with physical quantities: abundances as spatial maps and endmembers as spectral signatures.

Experimental results

Research questions

• RQ1 Can a non-data-driven deep learning architecture achieve superior spectral image recovery performance while maintaining interpretability?
• RQ2 How does incorporating both linear and non-linear low-rank mixture models improve spectral image reconstruction over standard deep image prior methods?
• RQ3 To what extent can learned endmembers and abundances from Mixture-Net be used directly for unmixing and material identification without additional routines?
• RQ4 Does enforcing low-rank structure at the input and within the network architecture enhance recovery quality in ill-posed spectral imaging problems?
• RQ5 Can the integration of model-based regularizers in a differentiable framework outperform purely data-driven baselines in spectral image recovery tasks?

Key findings

• Mixture-Net outperforms state-of-the-art data-driven and model-based methods in spectral image denoising, super-resolution, and compressive sensing across six benchmark datasets.
• The method achieves higher PSNR and SSIM values than competing approaches, with improvements of up to 1.5 dB in PSNR on real-world super-resolution data.
• The inclusion of the non-linearity block-layer significantly enhances reconstruction quality by modeling complex spectral interactions beyond linear mixing.
• Learned abundance maps and endmembers from Mixture-Net closely match ground-truth materials, enabling accurate unmixing without additional processing.
• The network's interpretability allows direct use of features for remote sensing tasks such as material identification, reducing reliance on post-processing routines.
• The non-data-driven nature of Mixture-Net eliminates the need for large-scale spectral image datasets, making it practical for real-world deployment.