[Paper Review] On the Effectiveness of Spectral Discriminators for Perceptual Quality Improvement

Xin Luo, Yunan Zhu+2 more|arXiv (Cornell University)|Jul 22, 2023

Image and Video Quality AssessmentComputer Science3 citations

TL;DR

This paper proposes a dual-stream discriminator combining spatial and spectral discriminators for GAN-based image super-resolution, enhancing perceptual quality by improving spectral alignment. By using a patch-wise Fourier transform and Transformer-based spectral aggregation (SpecFormer), the method achieves superior perceptual quality and more accurate no-reference image quality assessment compared to baselines.

ABSTRACT

Several recent studies advocate the use of spectral discriminators, which evaluate the Fourier spectra of images for generative modeling. However, the effectiveness of the spectral discriminators is not well interpreted yet. We tackle this issue by examining the spectral discriminators in the context of perceptual image super-resolution (i.e., GAN-based SR), as SR image quality is susceptible to spectral changes. Our analyses reveal that the spectral discriminator indeed performs better than the ordinary (a.k.a. spatial) discriminator in identifying the differences in the high-frequency range; however, the spatial discriminator holds an advantage in the low-frequency range. Thus, we suggest that the spectral and spatial discriminators shall be used simultaneously. Moreover, we improve the spectral discriminators by first calculating the patch-wise Fourier spectrum and then aggregating the spectra by Transformer. We verify the effectiveness of the proposed method twofold. On the one hand, thanks to the additional spectral discriminator, our obtained SR images have their spectra better aligned to those of the real images, which leads to a better PD tradeoff. On the other hand, our ensembled discriminator predicts the perceptual quality more accurately, as evidenced in the no-reference image quality assessment task.

Motivation & Objective

• To investigate the effectiveness of spectral discriminators in GAN-based image super-resolution (SR) from a frequency-domain perspective.
• To analyze the complementary strengths of spatial and spectral discriminators in detecting differences across low- and high-frequency ranges.
• To improve spectral discriminators by replacing 1D-MLP input with patch-wise Fourier transforms and Transformer-based aggregation.
• To validate that combining spatial and spectral discriminators leads to better perceptual quality and more accurate no-reference image quality assessment (NR-IQA).

Proposed method

• Proposes a dual discriminator architecture that jointly employs a spatial discriminator and a spectral discriminator to improve perceptual quality in GAN-based SR.
• Introduces SpecFormer, a Transformer-based spectral discriminator that computes patch-wise Fourier transforms and aggregates spectra via self-attention mechanisms.
• Uses azimuthal averaging of Fourier spectra to analyze frequency distribution differences between real and generated images.
• Employs a two-stream training strategy where spatial and spectral discriminators are trained separately but used in tandem during GAN optimization.
• Validates the method on image super-resolution and no-reference image quality assessment (NR-IQA), using metrics like PSNR, SRCC, PLCC, and KRCC.
• Adapts the RecycleD framework for NR-IQA, replacing its spatial discriminator with an ensemble of spatial and spectral discriminators to improve quality prediction.

Experimental results

Research questions

• RQ1 How do spectral and spatial discriminators compare in detecting differences in low- and high-frequency components of images?
• RQ2 Can spectral discriminators effectively compensate for the spatial discriminator’s tendency to favor high-frequency artifacts?
• RQ3 Does improving the spectral discriminator’s architecture via patch-wise Fourier transforms and Transformers lead to better perceptual quality in SR?
• RQ4 Can the ensemble of spatial and spectral discriminators improve no-reference image quality assessment performance?

Key findings

• The spectral discriminator outperforms the spatial discriminator in identifying differences in the high-frequency range, while the spatial discriminator performs better in the low-frequency range.
• The proposed SpecFormer, which uses patch-wise Fourier transforms and Transformer aggregation, improves spectral discriminator performance over 1D-MLP baselines.
• The dual discriminator (spatial + spectral) achieves the best perceptual quality in SR, with spectra of generated images better aligned to real images, improving the PD tradeoff.
• On the KonIQ-10k dataset, the VGG+SpecFormer combination achieved the highest SRCC of 0.6321 and PLCC of 0.6543 in NR-IQA, outperforming all baselines.
• The ensemble discriminator improved no-reference image quality assessment performance across all tested datasets, with the best results on both PIPAL and KonIQ-10k.
• Ablation studies confirm that SpecFormer consistently improves performance across various spatial discriminators, especially when combined with VGG.