[Paper Review] Multi-task convolutional neural network for image aesthetic assessment

Derya Soydaner, Johan Wagemans|arXiv (Cornell University)|May 16, 2023

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes a multi-task convolutional neural network that jointly predicts overall image aesthetic scores and multiple aesthetic attributes (e.g., composition, depth, semantics) using shared representations. The model achieves state-of-the-art performance on the AADB benchmark with near-human Spearman’s rank correlation (ρ = 0.707) and establishes a new baseline on the EVA dataset, outperforming prior methods while using fewer parameters.

ABSTRACT

As people's aesthetic preferences for images are far from understood, image aesthetic assessment is a challenging artificial intelligence task. The range of factors underlying this task is almost unlimited, but we know that some aesthetic attributes affect those preferences. In this study, we present a multi-task convolutional neural network that takes into account these attributes. The proposed neural network jointly learns the attributes along with the overall aesthetic scores of images. This multi-task learning framework allows for effective generalization through the utilization of shared representations. Our experiments demonstrate that the proposed method outperforms the state-of-the-art approaches in predicting overall aesthetic scores for images in one benchmark of image aesthetics. We achieve near-human performance in terms of overall aesthetic scores when considering the Spearman's rank correlations. Moreover, our model pioneers the application of multi-tasking in another benchmark, serving as a new baseline for future research. Notably, our approach achieves this performance while using fewer parameters compared to existing multi-task neural networks in the literature, and consequently makes our method more efficient in terms of computational complexity.

Motivation & Objective

• To develop a deep learning model that improves image aesthetic assessment by jointly learning overall aesthetic scores and key aesthetic attributes.
• To address the challenge of subjective and multifactorial aesthetic preferences through multi-task learning with shared representations.
• To evaluate the model’s performance and generalization across two major benchmarks: AADB and EVA.
• To demonstrate that multi-task learning improves performance and efficiency compared to single-task approaches.
• To establish a new state-of-the-art method for aesthetic score prediction with reduced model complexity.

Proposed method

• The model uses a shared convolutional backbone to extract hierarchical features from input RGB images.
• It employs multiple task-specific heads to predict both the overall aesthetic score and K individual aesthetic attribute scores.
• The loss function combines mean squared error for all regression tasks, optimizing for both overall and attribute scores simultaneously.
• Feature maps are visualized using Grad-CAM to interpret model attention and identify salient image regions influencing predictions.
• The architecture is trained end-to-end on paired data containing image-attribute and image-aesthetic score annotations.
• Cross-dataset evaluation is performed to assess generalization by training on one dataset and testing on the other.

Experimental results

Research questions

• RQ1 Can a multi-task CNN jointly learning aesthetic attributes and overall scores improve performance on image aesthetic assessment?
• RQ2 How does multi-task learning compare to single-task learning in terms of predictive accuracy and parameter efficiency?
• RQ3 Does the model generalize across different image aesthetic benchmarks, such as AADB and EVA?
• RQ4 To what extent do the predicted attribute scores align with human-annotated scores in terms of rank correlation?
• RQ5 Can the model achieve near-human performance in aesthetic prediction while maintaining low computational complexity?

Key findings

• The proposed multi-task CNN achieves a Spearman’s rank correlation of ρ = 0.707 on the AADB test set, setting a new state-of-the-art result.
• On the EVA dataset, the model achieves ρ = 0.695 in cross-dataset evaluation when trained on EVA and tested on AADB, outperforming the reverse scenario (ρ = 0.441).
• The model achieves the highest correlation (ρ = 0.97) for composition and depth attributes, closely aligning with human-annotated correlations on the EVA dataset.
• The model demonstrates superior performance with fewer parameters than previous multi-task approaches, supporting the principle of Occam’s razor in deep learning.
• The multi-task setting consistently outperforms single-task baselines across both datasets, confirming the benefit of shared representation learning.
• Grad-CAM visualizations confirm that the model focuses on semantically relevant image regions, such as composition and subject placement, for aesthetic judgments.