[论文解读] How intelligent are convolutional neural networks?

Zhennan Yan, Xiang Sean Zhou|arXiv (Cornell University)|Sep 18, 2017

Advanced Image and Video Retrieval Techniques参考文献 34被引用 10

一句话总结

本文通过测试深度卷积神经网络（DCNNs）从极少样本中学习类似格式塔的视觉概念（如对称性、计数、一致性）的能力，探究其语义理解能力。利用逐步增加复杂度的合成对抗性图像集，研究发现：人类在少量样本后通过一次‘顿悟时刻’即可实现近乎完美的泛化，而DCNNs则需要多数量级的数据仍无法在分布外或故意混淆的样本上表现良好，表明其缺乏真正的概念理解能力。

ABSTRACT

Motivated by the Gestalt pattern theory, and the Winograd Challenge for language understanding, we design synthetic experiments to investigate a deep learning algorithm's ability to infer simple (at least for human) visual concepts, such as symmetry, from examples. A visual concept is represented by randomly generated, positive as well as negative, example images. We then test the ability and speed of algorithms (and humans) to learn the concept from these images. The training and testing are performed progressively in multiple rounds, with each subsequent round deliberately designed to be more complex and confusing than the previous round(s), especially if the concept was not grasped by the learner. However, if the concept was understood, all the deliberate tests would become trivially easy. Our experiments show that humans can often infer a semantic concept quickly after looking at only a very small number of examples (this is often referred to as an "aha moment": a moment of sudden realization), and performs perfectly during all testing rounds (except for careless mistakes). On the contrary, deep convolutional neural networks (DCNN) could approximate some concepts statistically, but only after seeing many (x10^4) more examples. And it will still make obvious mistakes, especially during deliberate testing rounds or on samples outside the training distributions. This signals a lack of true "understanding", or a failure to reach the right "formula" for the semantics. We did find that some concepts are easier for DCNN than others. For example, simple "counting" is more learnable than "symmetry", while "uniformity" or "conformance" are much more difficult for DCNN to learn. To conclude, we propose an "Aha Challenge" for visual perception, calling for focused and quantitative research on Gestalt-style machine intelligence using limited training examples.

研究动机与目标

• 评估深度卷积神经网络（DCNNs）是否能实现人类水平的对称性、计数和一致性等视觉概念的语义理解。
• 比较DCNNs与人类在学习抽象视觉模式时的样本效率和泛化能力。
• 基于结构和统计复杂度，识别哪些视觉概念本质上更易被DCNNs与人类学习。
• 提出一个新的基准——‘顿悟挑战’（'Aha Challenge'）——用于使用极少训练数据评估机器学习模型的高级视觉智能。

提出的方法

• 为每个视觉概念（如对称性、计数、一致性）生成包含正负样本的合成数据集，确保在布局、大小和噪声方面可控变化。
• 分轮次进行训练和测试，每轮复杂度逐步提升，并引入旨在暴露概念理解错误的对抗性样本。
• 人类在与DCNNs相同的条件下接受测试，不提供额外提示，以衡量其在极少样本后的泛化能力。
• 通过测量达到高测试准确率所需的训练样本数量来量化学习速度，通过分布外测试集评估泛化能力。
• DCNNs使用标准反向传播进行端到端训练，性能在分布内和对抗性扰动测试数据上进行评估。
• 建立‘挑战’框架，邀请算法提交、人类研究结果和新测试设计，以基准化视觉智能。

实验结果

研究问题

• RQ1 当仅提供少量样本时，DCNNs能否以与人类相同的速度和准确度学习对称性和一致性等抽象视觉概念？
• RQ2 在人类经历‘顿悟时刻’后即可轻松分类的对抗性设计测试样本上，DCNNs的性能如何退化？
• RQ3 哪些视觉概念（如计数、对称性、一致性）更易于通过数据驱动方式被DCNNs学习，原因是什么？
• RQ4 增加训练数据量在多大程度上能缩小人类与DCNNs在语义视觉理解方面的‘智能差距’？
• RQ5 能否设计一种系统性基准，如‘顿悟挑战’，以评估并提升机器在视觉感知中的智能？

主要发现

• 人类在极少数训练样本后即可实现完美或近乎完美的分类准确率，通常在一次‘顿悟时刻’内完成。
• DCNNs达到与人类相当的性能所需训练样本数量约为人类的10^4倍。
• 即使经过大量训练，DCNNs在分布外和对抗性设计的测试样本上仍系统性地出错，表明其缺乏真正的概念理解。
• 简单概念如对象计数（1至3个对象）比复杂概念如对称性或一致性更易被DCNNs学习，后者需要显著更多的数据且泛化能力仍差。
• 研究发现DCNNs无法自主推断不变规则（如‘大小不重要’）——此类规则必须硬编码，凸显其推理能力的根本局限。
• 结果支持需要一个新的基准——‘顿悟挑战’——以聚焦于使用极少数据实现类人洞察和泛化的视觉人工智能研究。