[Paper Review] Composition-Aware Image Aesthetics Assessment

Dong Liu, Rohit Puri+2 more|arXiv (Cornell University)|Jul 25, 2019

Visual Attention and Saliency Detection43 references4 citations

TL;DR

This paper proposes RGNet, a composition-aware deep learning framework for image aesthetics assessment that models mutual dependencies among local image regions via a region composition graph and graph convolutional networks. By densely partitioning images into spatial regions, encoding their aesthetics-preserving features, and reasoning over a graph of region similarities, RGNet achieves state-of-the-art performance on AVA and AADB benchmarks, outperforming prior methods by up to 6.19% in accuracy.

ABSTRACT

Automatic image aesthetics assessment is important for a wide variety of applications such as on-line photo suggestion, photo album management and image retrieval. Previous methods have focused on mapping the holistic image content to a high or low aesthetics rating. However, the composition information of an image characterizes the harmony of its visual elements according to the principles of art, and provides richer information for learning aesthetics. In this work, we propose to model the image composition information as the mutual dependency of its local regions, and design a novel architecture to leverage such information to boost the performance of aesthetics assessment. To achieve this, we densely partition an image into local regions and compute aesthetics-preserving features over the regions to characterize the aesthetics properties of image content. With the feature representation of local regions, we build a region composition graph in which each node denotes one region and any two nodes are connected by an edge weighted by the similarity of the region features. We perform reasoning on this graph via graph convolution, in which the activation of each node is determined by its highly correlated neighbors. Our method naturally uncovers the mutual dependency of local regions in the network training procedure, and achieves the state-of-the-art performance on the benchmark visual aesthetics datasets.

Motivation & Objective

• To improve image aesthetics assessment by explicitly modeling the mutual dependencies of visual elements in an image.
• To address the limitation of existing CNN-based methods that fail to encode composition-aware features due to reliance on global image features.
• To develop an end-to-end deep learning architecture that integrates local feature encoding with graph-based reasoning for better aesthetics prediction.
• To demonstrate that modeling regional interdependencies through a learnable graph structure significantly boosts performance on benchmark datasets.

Proposed method

• The method partitions an input image into dense local regions using a spatial grid to capture fine-grained visual details and multi-scale context.
• It employs an aesthetics-preserving feature encoding network to extract features from each region that preserve visual quality and contextual information.
• A region composition graph is constructed where each node represents a local region and edges are weighted by cosine similarity of region features.
• Graph convolutional networks (GCNs) are applied to propagate and refine features across the graph, enabling reasoning over long-range regional dependencies.
• The entire architecture is trained end-to-end, allowing the model to learn composition-aware representations directly from raw images.
• The final aesthetics score is predicted from the aggregated features after graph convolution, with no additional post-processing.

Experimental results

Research questions

• RQ1 Can modeling mutual dependencies among local image regions improve image aesthetics assessment beyond global feature representations?
• RQ2 How effectively can a learnable region composition graph capture long-range spatial and visual relationships relevant to aesthetics?
• RQ3 Does incorporating graph-based reasoning into a CNN-based feature extractor lead to measurable performance gains on standard benchmarks?
• RQ4 To what extent do the learned region similarities align with human perception of visual harmony and composition?

Key findings

• RGNet achieves a Spearman’s rho of 0.7104 on the AADB test set, outperforming the previous state-of-the-art by 3.22% in accuracy.
• On the AVA dataset, RGNet improves classification accuracy by 6.19% over the MNA-CNN-Scene baseline, which also claims composition preservation.
• The feature similarity maps from the graph convolution layer show significantly larger and more coherent hot regions, indicating effective modeling of long-range dependencies.
• The model's performance gain is attributed to the graph convolution mechanism, which enhances feature representations by incorporating correlated regional information.
• Visualization confirms that the model learns to associate semantically and visually coherent regions (e.g., a rock and its adjacent features) as highly correlated, aligning with artistic composition principles.