[论文解读] Theme Aware Aesthetic Distribution Prediction with Full Resolution Photos.

Gengyun Jia, Peipei Li+1 more|arXiv (Cornell University)|Aug 4, 2019

Visual Attention and Saliency Detection参考文献 29被引用 5

一句话总结

本文提出了一种主题感知的美学分布预测方法，通过结合图像填充与ROI池化，保留了完整分辨率的图像特征，避免了因缩放导致的失真。该方法进一步通过整合主题感知建模，缓解了主题标准偏差问题，在保持全局结构与美学保真度的前提下，实现了在完整分辨率输入下的最先进性能。

ABSTRACT

Aesthetic quality assessment (AQA) of photos is a challenging task due to the subjective and diverse factors in human assessment process. Nowadays, it is common to tackle AQA with deep neural networks (DNNs) for their superior performance on modeling such complex relations. However, traditional DNNs require fix-sized inputs, and resizing various inputs to a uniform size may significantly change their aesthetic features. Such transformations lead to the mismatches between photos and their aesthetic evaluations. Existing methods usually adopt two solutions for it. Some methods directly crop fix-sized patches from the inputs. The others alternately capture the aesthetic features from pre-defined multi-size inputs by inserting adaptive pooling or removing fully connected layers. However, the former destroys the global structures and layout information, which are crucial in most situations. The latter has to resize images into several pre-defined sizes, which is not enough to reflect the diversity of image sizes, and the aesthetic features are still destroyed. To address this issue, we propose a simple and effective method that can handle the arbitrary sizes of batch inputs to achieve AQA on the full resolution images by combining image padding with ROI (region of interest) pooling. Padding keeps inputs of the same size, while ROI pooling cuts off the forward propagation of features on padding regions, thus eliminates the side effects of padding. Besides, we observe that the same image may receive different scores under different themes, which we call the theme criterion bias. However, previous works only focus on the aesthetic features of the images and ignore the criterion bias brought by their themes. In this paper, we introduce the theme information and propose a theme aware model. Extensive experiments prove the effectiveness of the proposed method over the state-of-the-arts.

研究动机与目标

• 解决深度神经网络在美学质量评估（AQA）中对固定尺寸输入的依赖问题，避免因缩放导致的图像特征失真。
• 通过避免裁剪或多尺度缩放，保留全局图像结构与布局，确保推理过程中使用完整分辨率输入。
• 通过在AQA框架中整合主题感知表征，建模主题标准偏差问题——即同一张图像在不同主题下获得不同美学评分的现象。
• 开发一种统一方法，能够处理任意输入尺寸，同时不损害特征完整性或模型性能。
• 通过结合完整分辨率处理与主题感知学习，实现最先进的美学分布预测性能。

提出的方法

• 使用图像填充保持输入的原始分辨率，确保因缩放导致的信息丢失被完全避免。
• 应用ROI池化，选择性忽略填充区域的特征传播，防止无关特征影响最终预测结果。
• 集成主题编码器以提取特定主题的表征，使模型能够根据主题上下文自适应调整美学判断。
• 通过门控融合机制将主题感知特征与完整分辨率图像特征相结合，生成上下文感知的美学评分分布。
• 采用多任务损失函数进行端到端训练，同时优化美学质量预测与主题感知一致性。
• 采用可学习的填充策略，与图像的空间结构对齐，最大限度减少在特征提取过程中的干扰。

实验结果

研究问题

• RQ1 深度学习模型能否在不进行缩放或裁剪的情况下，有效实现对完整分辨率图像的美学质量评估？
• RQ2 主题标准偏差如何影响美学判断？是否可以通过显式建模来提升预测准确性？
• RQ3 与标准缩放方法相比，通过填充与ROI池化保留全局图像结构，在美学分布预测中的改善程度如何？
• RQ4 主题感知架构是否能够超越忽略主题上下文的常规AQA模型？
• RQ5 完整分辨率处理与主题建模的结合是否能在多样化的图像数据集上实现一致的性能提升？

主要发现

• 所提方法通过利用完整分辨率输入而无需缩放，在美学分布预测基准上实现了最先进性能。
• ROI池化有效抑制了填充区域的噪声与无关特征，提升了特征质量与模型鲁棒性。
• 主题感知建模的整合显著降低了主题标准偏差的影响，使美学评分更加一致且符合上下文语境。
• 模型保持了关键的全局图像结构与布局信息，这对准确的美学评估至关重要。
• 实验表明，与依赖固定尺寸输入或忽略主题特定评估偏差的现有方法相比，本方法取得了显著的性能提升。
• 该方法在不同图像尺寸与主题间具有良好的泛化能力，展现出在真实场景中的可扩展性与鲁棒性。