[论文解读] ARTxAI: Explainable Artificial Intelligence Curates Deep Representation Learning for Artistic Images using Fuzzy Techniques

Javier Fumanal-Idocin, Javier Andreu-Pérez+3 more|arXiv (Cornell University)|Aug 29, 2023

Aesthetic Perception and AnalysisNeuroscience被引用 3

一句话总结

本文提出ARTxAI，一种可解释人工智能框架，通过将模糊聚类和模糊规则系统与CLIP及微调后的ResNet50特征相结合，增强艺术图像的深度表征学习。该框架相较非上下文感知模型，将分类性能提升最高达26%，并通过模糊规则实现深度特征到视觉和风格属性的可解释映射。

ABSTRACT

Automatic art analysis employs different image processing techniques to classify and categorize works of art. When working with artistic images, we need to take into account further considerations compared to classical image processing. This is because such artistic paintings change drastically depending on the author, the scene depicted, and their artistic style. This can result in features that perform very well in a given task but do not grasp the whole of the visual and symbolic information contained in a painting. In this paper, we show how the features obtained from different tasks in artistic image classification are suitable to solve other ones of similar nature. We present different methods to improve the generalization capabilities and performance of artistic classification systems. Furthermore, we propose an explainable artificial intelligence method to map known visual traits of an image with the features used by the deep learning model considering fuzzy rules. These rules show the patterns and variables that are relevant to solve each task and how effective is each of the patterns found. Our results show that our proposed context-aware features can achieve up to $6\%$ and $26\%$ more accurate results than other context- and non-context-aware solutions, respectively, depending on the specific task. We also show that some of the features used by these models can be more clearly correlated to visual traits in the original image than others.

研究动机与目标

• 通过利用上下文和视觉特征，提升艺术图像分类系统的泛化能力和性能。
• 通过引入可解释人工智能（XAI）技术，解决深度学习模型在艺术分析中的黑箱问题。
• 利用模糊规则系统，将深度特征映射到绘画中的已知视觉和风格特征。
• 评估深度特征在相似艺术分类任务间的可迁移性。
• 通过模糊隶属度和规则归纳，量化特定特征与艺术风格及视觉线索的相关性。

提出的方法

• 使用模糊C-均值（FCM）聚类，从艺术图像的文本注释中生成上下文嵌入。
• 调整一种基于模糊规则的聚类算法，以建模艺术风格与特征激活之间的关系。
• 整合CLIP特征以进行对比预训练，丰富视觉表征学习。
• 在艺术图像数据集上微调ResNet50，以提取用于下游分类的深度特征。
• 使用模糊规则分类器（FRBC）解释哪些深度特征与特定艺术风格最相关。
• 应用主成分分析（PCA）以可视化样本间特征主导性和聚类模式。

Figure 1 : Scheme of the two proposed classification frameworks using contextual features. Option (A) uses the contextual features to regularise the training of the CNN. The last layer of the network is replaced to fit the number of classes in the task. Option (B) appends the visual and contextual f

实验结果

研究问题

• RQ1 相较于非上下文感知模型，上下文感知特征在艺术图像分类中的深度学习模型性能提升程度如何？
• RQ2 模型学习到的哪些深度特征与绘画中已知的视觉和风格特征具有最强相关性？
• RQ3 模糊规则在多大程度上能将抽象的深度特征映射到可解释的视觉特征（如艺术风格或时期）？
• RQ4 同一组深度特征是否能在多个相关艺术分类任务中实现良好泛化？
• RQ5 从FCM导出的模糊隶属度函数如何增强艺术图像分析中上下文嵌入的可解释性？

主要发现

• 所提出的上下文感知模型相较非上下文感知解决方案最高提升6%准确率，且在特定任务上相较非上下文感知基线模型最高提升26%。
• 特征15在FRBC中表现出最高可解释性评分（DS = 0.6888），表明其与规则‘立体主义 IS 低 且 文艺复兴早期 IS 高 且 点彩派 IS 低’具有强相关性。
• 特征16在规则‘分析性立体主义 IS 低 且 原始艺术与原始主义 IS 中等 且 点彩派 IS 中等’中获得高规则支持度（DS = 0.4714）。
• FRBC对特征2的分类性能达到MCC = 0.5001，表明其对规则‘文艺复兴早期 IS 高 且 北方文艺复兴 IS 高’具有强预测能力。
• MTL-FCM框架优于其他配置，FRBC评估中特征14的MCC = 0.4666，特征16的MCC = 0.4714。
• PCA可视化结果证实，某些特征（如特征14）在样本中始终占主导地位，表明其与分类任务高度相关。

Figure 2 : Example of the extraction of the different descriptors for a Grad-CAM heatmap. (a) shows the gradient magnitude for each pixel. We choose the highest value from that image as the descriptor. (b) shows the pixels denoted as relevant because their value was higher than the average value in