[Paper Review] A Hypothesis for the Aesthetic Appreciation in Neural Networks

Xu Cheng, Xin Wang+3 more|arXiv (Cornell University)|Jul 31, 2021

Aesthetic Perception and Analysis68 references4 citations

TL;DR

This paper hypothesizes that aesthetic images enhance neural network performance by strengthening salient concepts and suppressing inessential ones. Using multi-variate interaction analysis in a pre-trained ResNet-18, the authors revise images to amplify high-impact feature interactions, resulting in perceptually more aesthetic outputs, verified through qualitative and probabilistic analysis of foreground/background sharpening/blurring effects.

ABSTRACT

This paper proposes a hypothesis for the aesthetic appreciation that aesthetic images make a neural network strengthen salient concepts and discard inessential concepts. In order to verify this hypothesis, we use multi-variate interactions to represent salient concepts and inessential concepts contained in images. Furthermore, we design a set of operations to revise images towards more beautiful ones. In experiments, we find that the revised images are more aesthetic than the original ones to some extent.

Motivation & Objective

• To investigate whether aesthetic appreciation in images stems from neural networks enhancing salient concepts and suppressing inessential ones.
• To develop a method for image revision that strengthens high-impact feature interactions while reducing low-impact ones.
• To verify the hypothesis by measuring perceptual aesthetic improvement through controlled image modifications.
• To model aesthetic quality as a function of multi-variate interactions in deep neural network activations.
• To explore the role of local vs. global context in aesthetic perception using DNN-based image editing.

Proposed method

• The method uses multi-variate interaction measures from game theory to quantify the contribution of pixel groups to the DNN's classification confidence.
• Salient concepts are identified as pixel subsets S with large |I(S)|, while inessential concepts have negligible |I(S)| values.
• Image revision is performed by applying smooth, non-adversarial operations to adjust hue, saturation, brightness, and sharpness based on interaction gradients.
• The loss function is computed using a pre-trained ResNet-18 on ImageNet to guide revisions toward enhanced saliency and reduced inessentiality.
• Foreground and background regions are separated using object bounding boxes to compute probability distributions of blur/sharp scores.
• The method distinguishes small-scale (local) and large-scale (global) interactions by varying the receptive field size r in the interaction computation.

Experimental results

Research questions

• RQ1 Does enhancing high-impact multi-variate interactions in deep neural activations lead to more aesthetically pleasing images?
• RQ2 Can image revision based on saliency of feature interactions improve perceived aesthetic quality without introducing adversarial artifacts?
• RQ3 How do small-scale (local) and large-scale (global) interactions differ in their contribution to aesthetic perception?
• RQ4 Is there a measurable shift in pixel-level blur/sharpness that correlates with improved aesthetic judgment?
• RQ5 To what extent does suppressing inessential concepts (low |I(S)|) and amplifying salient ones (high |I(S)|) align with human aesthetic preferences?

Key findings

• Penalizing small-scale concepts (r_small ∈ {0.1n, 0.2n}) led to sharpening of foreground objects and blurring of backgrounds, aligning with common photographic aesthetic principles.
• Punishing large-scale concepts (r_large ∈ {0.8n, 0.9n}) resulted in the opposite effect—sharpening the background and blurring the foreground—indicating different roles of local and global context.
• The probability P(F|Δ_pixel) exceeded 0.5 for pixels with negative Δ_pixel (blurring), indicating that blurring was more likely to occur in the foreground when small-scale interactions were penalized.
• The revised images were perceived as more aesthetic than the originals, supporting the hypothesis that aesthetic quality correlates with enhanced saliency and suppressed inessentiality in neural representations.
• The method successfully avoided introducing out-of-distribution artifacts, as it used smooth, non-adversarial image transformations.
• The decomposition of network utility into multi-variate interaction terms I(S) revealed that only a sparse subset of pixel groups significantly influenced classification confidence.