[Paper Review] GANs-NQM: A Generative Adversarial Networks based No Reference Quality Assessment Metric for RGB-D Synthesized Views

Suiyi Ling, Jing Li+2 more|arXiv (Cornell University)|Mar 28, 2019

Advanced Vision and Imaging54 references6 citations

TL;DR

This paper proposes GANs-NQM, a no-reference quality metric for RGB-D synthesized views using a generative adversarial network (GAN) trained on large-scale 2D datasets instead of RGB-D data. The method leverages the trained discriminator to learn a 'Bag of Distortion Words' codebook and identify local distortions, enabling accurate perceptual quality prediction that outperforms state-of-the-art metrics while also producing a high-quality context inpainter as a byproduct.

ABSTRACT

In this paper, we proposed a no-reference (NR) quality metric for RGB plus image-depth (RGB-D) synthesis images based on Generative Adversarial Networks (GANs), namely GANs-NQM. Due to the failure of the inpainting on dis-occluded regions in RGB-D synthesis process, to capture the non-uniformly distributed local distortions and to learn their impact on perceptual quality are challenging tasks for objective quality metrics. In our study, based on the characteristics of GANs, we proposed i) a novel training strategy of GANs for RGB-D synthesis images using existing large-scale computer vision datasets rather than RGB-D dataset; ii) a referenceless quality metric based on the trained discriminator by learning a `Bag of Distortion Word' (BDW) codebook and a local distortion regions selector; iii) a hole filling inpainter, i.e., the generator of the trained GANs, for RGB-D dis-occluded regions as a side outcome. According to the experimental results on IRCCyN/IVC DIBR database, the proposed model outperforms the state-of-the-art quality metrics, in addition, is more applicable in real scenarios. The corresponding context inpainter also shows appealing results over other inpainting algorithms.

Motivation & Objective

• To address the challenge of evaluating perceptual quality in RGB-D synthesized views, particularly due to non-uniform inpainting distortions in dis-occluded regions.
• To develop a no-reference quality metric that does not require a reference image, enabling real-time and practical deployment.
• To overcome the limitation of small-scale, noisy RGB-D datasets by training on large-scale 2D computer vision datasets.
• To simultaneously generate a high-performing context inpainter as a side benefit of the GAN training process.
• To model perceptually relevant distortions such as blurriness and ghosting in complex texture regions and object boundaries.

Proposed method

• A novel GAN training strategy is proposed that uses only 2D image datasets (e.g., ImageNet) instead of RGB-D datasets, with synthetic depth maps and masking to simulate dis-occlusion artifacts.
• The discriminator is fine-tuned to learn a 'Bag of Distortions Words' (BDW) codebook that encodes local distortion patterns from synthesized views.
• A local distortion region selector is derived from the discriminator’s feature maps to identify and localize non-uniform inpainting artifacts.
• The BDW codebook and distortion region features are used as input to a support vector regressor (SVR) to predict perceptual quality scores.
• The generator of the trained GAN is repurposed as a context inpainter for dis-occluded regions in RGB-D synthesis, leveraging the same training data and architecture.
• The model is trained using a combination of perceptual loss and adversarial loss to preserve structural and textural fidelity in hole-filling.

Experimental results

Research questions

• RQ1 Can a GAN trained on large-scale 2D image datasets effectively learn to assess perceptual quality in RGB-D synthesized views without requiring reference images or RGB-D datasets?
• RQ2 How can the discriminator of a GAN be leveraged to detect and quantify non-uniform inpainting distortions in dis-occluded regions of synthesized views?
• RQ3 Can the generator of such a GAN be effectively repurposed as a high-performance context inpainter for RGB-D synthesis?
• RQ4 How does the proposed GANs-NQM metric compare in performance to existing state-of-the-art full-reference and no-reference quality metrics on real-world RGB-D synthesis data?
• RQ5 To what extent does the proposed training strategy, which avoids noisy depth data, generalize to real-world dis-occlusion artifacts?

Key findings

• GANs-NQM achieves the highest performance among all evaluated metrics on the IRCCyN/IVC DIBR database, outperforming both full-reference and no-reference state-of-the-art methods.
• The model’s normalized execution time is 1.25, which is significantly faster than the second-best NR metric APT and the best FR metric ST-IQM, indicating strong real-time applicability.
• The pre-trained generator (inpainter) produces superior results on challenging dis-occluded regions—especially at object boundaries and in homogeneous texture areas—compared to other inpainting algorithms.
• The inpainter preserves fine structures such as hair strands and poster corners more accurately, reducing ghosting and double-edge artifacts.
• The use of 2D datasets instead of RGB-D datasets enables effective training with only 10K images, demonstrating scalability and robustness despite limited data.
• The BDW codebook and local distortion selector effectively capture perceptually salient artifacts, enabling accurate quality prediction even in complex, non-uniform distortion patterns.