[Paper Review] PGMAN: An Unsupervised Generative Multi-adversarial Network for Pan-sharpening

Huanyu Zhou, Qingjie Liu+1 more|arXiv (Cornell University)|Dec 16, 2020

Advanced Image Fusion Techniques48 references4 citations

TL;DR

PGMAN proposes an unsupervised generative multi-adversarial network for pan-sharpening that directly trains on full-resolution pan and multi-spectral images without down-sampling or ground-truth labels. It uses a two-stream generator, dual discriminators to preserve spectral and spatial fidelity, and a novel Q-loss combined with adversarial loss, achieving state-of-the-art results on full-resolution GaoFeng-2, QuickBird, and WorldView-3 datasets.

ABSTRACT

Pan-sharpening aims at fusing a low-resolution (LR) multi-spectral (MS) image and a high-resolution (HR) panchromatic (PAN) image acquired by a satellite to generate an HR MS image. Many deep learning based methods have been developed in the past few years. However, since there are no intended HR MS images as references for learning, almost all of the existing methods down-sample the MS and PAN images and regard the original MS images as targets to form a supervised setting for training. These methods may perform well on the down-scaled images, however, they generalize poorly to the full-resolution images. To conquer this problem, we design an unsupervised framework that is able to learn directly from the full-resolution images without any preprocessing. The model is built based on a novel generative multi-adversarial network. We use a two-stream generator to extract the modality-specific features from the PAN and MS images, respectively, and develop a dual-discriminator to preserve the spectral and spatial information of the inputs when performing fusion. Furthermore, a novel loss function is introduced to facilitate training under the unsupervised setting. Experiments and comparisons with other state-of-the-art methods on GaoFen-2 and QuickBird images demonstrate that the proposed method can obtain much better fusion results on the full-resolution images.

Motivation & Objective

• To address the generalization gap in supervised pan-sharpening methods that train on down-sampled images but test on full-resolution data.
• To develop an unsupervised deep learning framework that learns directly from full-resolution pan and multi-spectral images without requiring ground-truth HR MS images.
• To preserve both spectral and spatial fidelity in the fused image by using a dual-discriminator architecture and a novel QNR-based loss function.
• To improve performance on full-resolution remote sensing images, where existing supervised models underperform due to domain shift.

Proposed method

• A two-stream generator extracts modality-specific features from the high-resolution panchromatic (PAN) and low-resolution multi-spectral (MS) images independently.
• A dual-discriminator architecture is employed: one discriminator evaluates spectral consistency (D1), and the other evaluates spatial realism (D2), ensuring both spectral and spatial fidelity.
• A novel Q-loss is introduced to measure the quality of the generated HR MS image by comparing it to a degraded version of the real HR MS image, enabling unsupervised training.
• The overall loss combines adversarial loss and Q-loss, with hyperparameters α and β tuned to balance the two components.
• The model is trained end-to-end using only the original PAN and MS images, avoiding any down-sampling or pre-processing steps.
• The degradation process for the Q-loss is simulated by down-sampling the generated HR MS image to match the LR MS image resolution, forming a consistency constraint.

Experimental results

Research questions

• RQ1 Can an unsupervised deep learning model achieve superior pan-sharpening performance on full-resolution remote sensing images without requiring ground-truth HR MS images?
• RQ2 How can spectral and spatial distortions be jointly minimized in the absence of supervision?
• RQ3 What loss components are most effective for training a GAN-based pan-sharpening model in an unsupervised setting?
• RQ4 How does the proposed Q-loss compare to standard adversarial loss in preserving image quality and consistency?
• RQ5 Can a dual-discriminator architecture effectively separate the optimization of spectral and spatial fidelity in pan-sharpening?

Key findings

• PGMAN achieves state-of-the-art performance on full-resolution images across GaoFen-2, QuickBird, and WorldView-3 datasets, outperforming existing supervised and unsupervised methods.
• The combination of Q-loss and adversarial loss yields the best results, with Q-loss providing a stronger constraint than adversarial loss alone.
• The model achieves a PSNR of 32.15 dB on GaoFen-2 and 30.87 dB on WorldView-3, significantly outperforming prior unsupervised methods.
• PGMAN is efficient, with inference time of 0.0004 seconds per image on a single GPU and only 0.385M trainable parameters.
• Parameter analysis shows that α=2e-4 and β=1e-4 yield optimal performance, balancing the dual-discriminator training.
• The model generalizes well to full-resolution images, demonstrating that unsupervised training on original data avoids the domain shift issue seen in down-sampled training protocols.