[Paper Review] Full RGB Just Noticeable Difference (JND) Modelling

Jian Jin, Dong Yu+4 more|arXiv (Cornell University)|Mar 1, 2022

Color Science and Applications4 citations

TL;DR

This paper proposes RGB-JND-NET, a deep learning-based generative model for full RGB Just Noticeable Difference (JND) modelling that jointly considers luminance and chrominance channels, visual attention, and adaptive image quality assessment. It achieves state-of-the-art performance by learning perceptual redundancy in full color space, with experimental results showing higher JND tolerance in red and blue channels compared to green, consistent with human visual system sensitivity.

ABSTRACT

Just Noticeable Difference (JND) has many applications in multimedia signal processing, especially for visual data processing up to date. It's generally defined as the minimum visual content changes that the human can perspective, which has been studied for decades. However, most of the existing methods only focus on the luminance component of JND modelling and simply regard chrominance components as scaled versions of luminance. In this paper, we propose a JND model to generate the JND by taking the characteristics of full RGB channels into account, termed as the RGB-JND. To this end, an RGB-JND-NET is proposed, where the visual content in full RGB channels is used to extract features for JND generation. To supervise the JND generation, an adaptive image quality assessment combination (AIC) is developed. Besides, the RDB-JND-NET also takes the visual attention into account by automatically mining the underlying relationship between visual attention and the JND, which is further used to constrain the JND spatial distribution. To the best of our knowledge, this is the first work on careful investigation of JND modelling for full-color space. Experimental results demonstrate that the RGB-JND-NET model outperforms the relevant state-of-the-art JND models. Besides, the JND of the red and blue channels are larger than that of the green one according to the experimental results of the proposed model, which demonstrates that more changes can be tolerated in the red and blue channels, in line with the well-known fact that the human visual system is more sensitive to the green channel in comparison with the red and blue ones.

Motivation & Objective

• To address the limitation of existing JND models that treat chrominance as scaled luminance, this work investigates full-color space JND modelling.
• To develop a learning-based generative network that accurately models perceptual redundancy across all three RGB channels.
• To improve JND generation by integrating visual attention as a spatial constraint, rather than using it as simple weighting.
• To overcome the limitations of fixed IQA metrics in training by introducing an adaptive IQA combination (AIC) that dynamically selects optimal quality assessment metrics.
• To validate the model's perceptual accuracy through subjective and objective evaluations, demonstrating improved JND estimation in full RGB space.

Proposed method

• Proposes RGB-JND-NET, a deep neural network that takes full RGB input to extract multi-scale features for JND prediction.
• Introduces an adaptive image quality assessment combination (AIC) that dynamically selects from multiple IQA metrics (e.g., SSIM, LPIPS, etc.) to supervise JND generation and avoid perceptual degradation.
• Incorporates visual attention maps as a spatial constraint, where the network learns the implicit relationship between attention and JND distribution to improve spatial accuracy.
• Uses a gradient-based magnitude constraint on full RGB channels to ensure generated JNDs remain perceptually undetectable.
• Employs an auto-encoder-like architecture with skip connections to preserve spatial details and enhance reconstruction fidelity.
• Trains the model end-to-end using a combination of perceptual loss and AIC-based distortion supervision to maintain visual quality.

Experimental results

Research questions

• RQ1 How can JND modelling be improved by considering full RGB color channels instead of treating chrominance as a scaled version of luminance?
• RQ2 What is the impact of visual attention on the spatial distribution of JND, and how can it be effectively integrated into the JND generation process?
• RQ3 Can an adaptive IQA combination (AIC) outperform fixed IQA metrics in supervising generative JND models?
• RQ4 How does the proposed model perform across different image content types in terms of JND distribution across RGB channels?
• RQ5 To what extent does the model reflect known human visual system (HVS) sensitivities, such as higher sensitivity to green compared to red and blue?

Key findings

• The proposed RGB-JND-NET model outperforms existing state-of-the-art JND models in both subjective and objective evaluations.
• The model generates higher JND values in the red and blue channels compared to the green channel, confirming the human visual system's greater sensitivity to green.
• The AIC technique successfully identifies and suppresses perceptually harmful distortions during training, leading to higher quality JND outputs.
• The integration of visual attention as a learned feature improves spatial consistency of JND distribution, especially in salient regions.
• The model adapts JND distribution based on image content, showing different patterns across images with varying visual complexity.
• The dominant distortion type in generated JNDs is JPEG/JPEG2000-like compression, indicating the model learns perceptually plausible distortions.