[Paper Review] Learning Image Aesthetic Assessment from Object-level Visual Components

Jingwen Hou, Sheng Yang+3 more|arXiv (Cornell University)|Apr 4, 2021

Visual Attention and Saliency Detection44 references4 citations

TL;DR

This paper proposes IAA-OVC, a novel deep learning framework for image aesthetic assessment that models aesthetics through object-level visual components (OVCs) detected by a generic object detector. By applying object-level and graph attention mechanisms to dynamically weigh individual OVCs and their pairwise relationships, the model improves performance on aesthetic rating distribution prediction and offers the first interpretable analysis of how visual components contribute to aesthetic judgments, aligning with photography and psychological theories.

ABSTRACT

As it is said by Van Gogh, great things are done by a series of small things brought together. Aesthetic experience arises from the aggregation of underlying visual components. However, most existing deep image aesthetic assessment (IAA) methods over-simplify the IAA process by failing to model image aesthetics with clearly-defined visual components as building blocks. As a result, the connection between resulting aesthetic predictions and underlying visual components is mostly invisible and hard to be explicitly controlled, which limits the model in both performance and interpretability. This work aims to model image aesthetics from the level of visual components. Specifically, object-level regions detected by a generic object detector are defined as visual components, namely object-level visual components (OVCs). Then generic features representing OVCs are aggregated for the aesthetic prediction based upon proposed object-level and graph attention mechanisms, which dynamically determines the importance of individual OVCs and relevance between OVC pairs, respectively. Experimental results confirm the superiority of our framework over previous relevant methods in terms of SRCC and PLCC on the aesthetic rating distribution prediction. Besides, quantitative analysis is done towards model interpretation by observing how OVCs contribute to aesthetic predictions, whose results are found to be supported by psychology on aesthetics and photography rules. To the best of our knowledge, this is the first attempt at the interpretation of a deep IAA model.

Motivation & Objective

• To address the lack of interpretability and component-level modeling in existing deep image aesthetic assessment (IAA) methods.
• To model image aesthetics not from holistic image features, but from predefined object-level visual components (OVCs) as building blocks.
• To enable dynamic, attention-based aggregation of OVC features based on their individual importance and interrelationships.
• To provide the first quantitative interpretation of deep IAA models by linking OVC contributions to aesthetic predictions.
• To validate the model’s alignment with photography rules (e.g., attention-subject consistency) and psychological theories (e.g., visual rightness).

Proposed method

• Object-level visual components (OVCs) are defined as object detections from a generic object detector (e.g., Faster R-CNN), serving as the fundamental visual units.
• Each OVC is represented by deep generic features (e.g., MLSP features) extracted from its region, forming the input to the model.
• An Object-level Attention-based Re-weighting module computes dynamic importance scores for each OVC, mimicking subject detection and attention allocation.
• A Graph Attention-based Aggregation module computes pairwise relevance scores between OVCs, modeling the saliency of object organizations.
• The final aesthetic representation is formed by aggregating OVC features using learned attention weights, followed by a regression head for rating distribution prediction.
• Interpretability is achieved through correlation analysis between OVC attention scores and aesthetic scores, validated on both training and test sets.

Experimental results

Research questions

• RQ1 Can modeling image aesthetics through object-level visual components (OVCs) improve performance and interpretability in deep IAA models?
• RQ2 Do learned object-level attention scores correlate with subject detection and attention-subject consistency in photography?
• RQ3 Is the relevance between OVC pairs (via graph attention) predictive of aesthetic quality, as suggested by the visual rightness theory in psychology?
• RQ4 Can the attention patterns learned during training generalize to unseen test images, indicating robustness and generalization of learned rules?
• RQ5 Do the model's learned attention patterns align with established photography rules and psychological principles of aesthetic perception?

Key findings

• The proposed IAA-OVC framework achieves superior performance on the AVA dataset, outperforming prior methods in both SRCC and PLCC for aesthetic rating distribution prediction.
• Object-level attention scores show strong correlation with subject presence: higher attention on subjects like 'face' or 'person' correlates positively with higher aesthetic scores, while attention on non-subjects like 'ear' correlates negatively.
• Blurry regions receiving high attention are strongly negatively correlated with aesthetic scores, confirming that image distortions undermine aesthetic quality.
• Graph attention scores for object class pairs (e.g., 'eye' and 'face') show strong positive correlation with aesthetic scores, indicating that salient object organizations enhance perceived quality.
• The correlation between attention patterns and aesthetic scores is consistently preserved from training to test sets, indicating robust and generalizable attention mechanisms.
• The model’s attention behavior aligns with psychological theories: subjects are prioritized, and salient object pairings (e.g., facial features) are emphasized, supporting the visual rightness and attention-subject consistency principles.