[Paper Review] A Survey of Hand Crafted and Deep Learning Methods for Image Aesthetic Assessment.

Saira Kanwal, Muhammad Uzair+1 more|arXiv (Cornell University)|Mar 22, 2021

Visual Attention and Saliency Detection95 references4 citations

TL;DR

This survey reviews hand-crafted and deep learning methods for automatic image aesthetic assessment, analyzing features like balance, contrast, and harmony to classify images into aesthetic levels. It compares performance across methods and identifies limitations, offering insights into why certain models or features outperform others.

ABSTRACT

Automatic image aesthetics assessment is a computer vision problem that deals with the categorization of images into different aesthetic levels. The categorization is usually done by analyzing an input image and computing some measure of the degree to which the image adhere to the key principles of photography (balance, rhythm, harmony, contrast, unity, look, feel, tone and texture). Owing to its diverse applications in many areas, automatic image aesthetic assessment has gained significant research attention in recent years. This paper presents a literature review of the recent techniques of automatic image aesthetics assessment. A large number of traditional hand crafted and deep learning based approaches are reviewed. Key problem aspects are discussed such as why some features or models perform better than others and what are the limitations. A comparison of the quantitative results of different methods is also provided at the end.

Motivation & Objective

• To provide a comprehensive review of recent techniques in automatic image aesthetic assessment.
• To analyze the strengths and limitations of hand-crafted and deep learning-based approaches.
• To compare the quantitative performance of different methods across benchmark datasets.
• To identify why certain features or models perform better than others in aesthetic classification.
• To highlight key challenges and open problems in the field of image aesthetic assessment.

Proposed method

• Systematic literature review of recent works in automatic image aesthetic assessment from peer-reviewed journals and conferences.
• Categorization of methods into hand-crafted feature-based and deep learning-based approaches.
• Analysis of key photographic principles such as balance, rhythm, harmony, contrast, unity, tone, and texture as underlying features.
• Evaluation of model performance using standard metrics on benchmark datasets, including correlation with human judgments.
• Comparison of results across methods in terms of correlation coefficients (e.g., Pearson and Spearman) and mean squared error.
• Identification of recurring patterns in feature engineering and network architecture that contribute to improved performance.

Experimental results

Research questions

• RQ1 What are the most effective hand-crafted features for image aesthetic assessment, and how do they relate to photographic principles?
• RQ2 How do deep learning models compare to traditional hand-crafted methods in terms of performance and generalization?
• RQ3 Why do certain models or feature sets achieve higher correlation with human aesthetic judgments?
• RQ4 What are the key limitations and challenges in current automatic image aesthetic assessment methods?
• RQ5 Which datasets and evaluation metrics are most commonly used, and how do they influence reported performance?

Key findings

• Deep learning-based methods generally outperform hand-crafted feature approaches in terms of correlation with human judgments.
• Features related to composition, contrast, and texture are consistently important across multiple models and datasets.
• Models that incorporate both low-level and high-level features tend to achieve better performance than those relying on single-level features.
• Despite progress, significant performance gaps remain between state-of-the-art models and human-level aesthetic judgment.
• The choice of evaluation metric and benchmark dataset strongly influences reported results, suggesting caution in model comparison.
• Limitations include overfitting to specific datasets, lack of interpretability in deep models, and inconsistent generalization across diverse image categories.