Skip to main content
QUICK REVIEW

[论文解读] AesBench: An Expert Benchmark for Multimodal Large Language Models on Image Aesthetics Perception

Yipo Huang, Quan Yuan|arXiv (Cornell University)|Jan 16, 2024
Multimodal Machine Learning ApplicationsComputer Science被引用 6
一句话总结

AesBench 构建了一个专家标注的美学数据库和一个四维评估框架,用于衡量多模态大语言模型在图像美感感知、共情、评估和解读方面的能力。

ABSTRACT

With collective endeavors, multimodal large language models (MLLMs) are undergoing a flourishing development. However, their performances on image aesthetics perception remain indeterminate, which is highly desired in real-world applications. An obvious obstacle lies in the absence of a specific benchmark to evaluate the effectiveness of MLLMs on aesthetic perception. This blind groping may impede the further development of more advanced MLLMs with aesthetic perception capacity. To address this dilemma, we propose AesBench, an expert benchmark aiming to comprehensively evaluate the aesthetic perception capacities of MLLMs through elaborate design across dual facets. (1) We construct an Expert-labeled Aesthetics Perception Database (EAPD), which features diversified image contents and high-quality annotations provided by professional aesthetic experts. (2) We propose a set of integrative criteria to measure the aesthetic perception abilities of MLLMs from four perspectives, including Perception (AesP), Empathy (AesE), Assessment (AesA) and Interpretation (AesI). Extensive experimental results underscore that the current MLLMs only possess rudimentary aesthetic perception ability, and there is still a significant gap between MLLMs and humans. We hope this work can inspire the community to engage in deeper explorations on the aesthetic potentials of MLLMs. Source data will be available at https://github.com/yipoh/AesBench.

研究动机与目标

  • Motivate evaluation of multimodal large language models (MLLMs) on the abstract task of image aesthetics perception.
  • Construct a high-quality expert-annotated dataset (EAPD) covering diverse image types and aesthetics.
  • Propose a four-dimensional evaluation framework: Perception (AesP), Empathy (AesE), Assessment (AesA), and Interpretation (AesI).
  • Assess 15 MLLMs (including GPT-4V and Gemini Pro Vision) to reveal gaps between current models and human performance.
  • Provide publicly accessible code and data to spur further research into aesthetic capabilities of MLLMs.

提出的方法

  • Create the Expert-labeled Aesthetics Perception Database (EAPD) with 2,800 images and 8,400 question–answer pairs plus 2,800 aesthetic interpretations, annotated by professional aesthetics experts.
  • Divide EAPD into four subsets corresponding to benchmark objectives: AesPQA, AesEQA, AesAQA, and AesInter.
  • Define a four-dimensional evaluation framework: AesP (perception of aesthetic attributes), AesE (empathy and emotional resonance), AesA (aesthetic quality assessment), and AesI (interpretation of aesthetic reasons).
  • Use yes/no, what, how, and why question types to test each dimension, and apply a GPT-assisted evaluation strategy to handle output format variability.
  • Evaluate 15 MLLMs with original weights (no dataset-specific tuning) and use a GPT-based ground-truth check for outputs not conforming to option formats.

实验结果

研究问题

  • RQ1Can current MLLMs correctly identify aesthetic attributes in images (AesP)?
  • RQ2Can MLLMs understand and reflect emotional and experiential aspects of aesthetics (AesE)?
  • RQ3How accurately can MLLMs classify or rate image aesthetics (AesA)?
  • RQ4To what extent can MLLMs generate coherent, expert-aligned interpretations of aesthetic quality (AesI)?

主要发现

  • AesBench distinguishes which MLLMs excel at different aesthetic tasks, revealing substantial gaps relative to humans.
  • GPT-4V and Gemini Pro Vision perform strong in perception and empathy tasks, but overall accuracy remains limited across image types (NIs, AIs, AGIs).
  • AesA results show maximal accuracy around 52.86% for the best model (Q-Instruct), indicating limited automated aesthetic scoring capability.
  • Aesthetic interpretation shows high relevance and completeness but relatively low precision, suggesting hallucination risks in language-based explanations.
  • Across dimensions, natural images are easier for models than artistic or AI-generated images, reflecting training data biases.
  • The benchmark emphasizes the need for improved precision and interpretation quality in MLLMs’ aesthetic reasoning.

阅读本论文

在 Nubint 阅读器中打开原文,边读边就任意段落提问。

在 Nubint 中阅读本论文

本解读由 AI 根据论文原文生成,引用前请核对原文。