[论文解读] Visual Concepts and Compositional Voting

Jianyu Wang, Zhishuai Zhang+6 more|arXiv (Cornell University)|Nov 13, 2017

Domain Adaptation and Few-Shot Learning参考文献 38被引用 16

一句话总结

本文提出了一种组合投票框架，利用从车辆图像的深度网络激活中提取的视觉概念作为可解释的构建模块，用于模式理论视觉模型。通过聚类预训练网络的特征图，作者识别出语义上有意义的视觉概念，这些概念构成图像模式的稀疏符号编码，从而实现一种生成模型，该模型在遮挡条件下，尤其是在上下文具有对抗性时，优于深度网络和SVM。

ABSTRACT

It is very attractive to formulate vision in terms of pattern theory \cite{Mumford2010pattern}, where patterns are defined hierarchically by compositions of elementary building blocks. But applying pattern theory to real world images is currently less successful than discriminative methods such as deep networks. Deep networks, however, are black-boxes which are hard to interpret and can easily be fooled by adding occluding objects. It is natural to wonder whether by better understanding deep networks we can extract building blocks which can be used to develop pattern theoretic models. This motivates us to study the internal representations of a deep network using vehicle images from the PASCAL3D+ dataset. We use clustering algorithms to study the population activities of the features and extract a set of visual concepts which we show are visually tight and correspond to semantic parts of vehicles. To analyze this we annotate these vehicles by their semantic parts to create a new dataset, VehicleSemanticParts, and evaluate visual concepts as unsupervised part detectors. We show that visual concepts perform fairly well but are outperformed by supervised discriminative methods such as Support Vector Machines (SVM). We next give a more detailed analysis of visual concepts and how they relate to semantic parts. Following this, we use the visual concepts as building blocks for a simple pattern theoretical model, which we call compositional voting. In this model several visual concepts combine to detect semantic parts. We show that this approach is significantly better than discriminative methods like SVM and deep networks trained specifically for semantic part detection. Finally, we return to studying occlusion by creating an annotated dataset with occlusion, called VehicleOcclusion, and show that compositional voting outperforms even deep networks when the amount of occlusion becomes large.

研究动机与目标

• 探究深度网络的内部表征是否可被解释为用于模式理论建模的有意义的视觉概念。
• 开发一种无需监督即可从深度特征中提取语义一致的视觉概念的方法。
• 评估这些视觉概念能否作为组合模型的构建模块，以在遮挡条件下提升泛化能力。
• 检验假设：基于视觉概念的组合模型在对抗性遮挡测试下是否优于SVM和深度网络等判别模型。
• 创建新数据集——VehicleSemanticParts和VehicleOcclusion，以支持对部件检测和遮挡鲁棒性的严格评估。

提出的方法

• 在PASCAL3D+车辆数据集上，对预训练深度网络的高层特征图（例如，第4层）进行聚类，以提取视觉概念。
• 对512维特征向量使用k-means聚类，识别出视觉紧凑且语义有意义的视觉概念。
• 定义基于阈值的稀疏编码：若某像素的特征距离其概念质心低于学习得到的阈值（Tm ≈ 0.7），则该像素被该视觉概念编码。
• 基于激活超过阈值的视觉概念，为每个图像块构建二值激活向量，从而实现空间模式的符号化表示。
• 构建组合投票模型，通过二值概念激活上的因子化生成模型，使多个视觉概念组合以检测语义部件。
• 在新数据集VehicleOcclusion上评估模型，通过随机添加遮挡物模拟现实复杂性，以测试其鲁棒性。

实验结果

研究问题

• RQ1 从深度网络激活中提取的视觉概念能否作为模式理论视觉模型的可解释、语义上有意义的构建模块？
• RQ2 与有监督的部件检测器相比，这些视觉概念在准确性和鲁棒性方面表现如何？
• RQ3 基于视觉概念的组合模型是否能在遮挡条件下优于SVM和深度网络等判别模型？
• RQ4 稀疏编码特征为视觉概念的最优阈值是什么？其对模式表征有何影响？
• RQ5 在未遮挡数据上训练的组合投票模型，在对抗性遮挡测试下是否比深度网络泛化能力更强？

主要发现

• 通过聚类深度特征提取出的视觉概念在视觉上紧凑，并与车辆的语义部件相对应，该结论已通过VehicleSemanticParts数据集中的人工标注真实值验证。
• 无监督提取的视觉概念作为部件检测器表现合理，但在未遮挡数据上仍逊于有监督的SVM。
• 组合投票模型通过组合多个视觉概念来检测部件，在未遮挡数据上显著优于SVM和深度网络。
• 在对抗性遮挡条件下，组合投票模型显著优于深度网络，后者在上下文被遮挡物破坏时性能迅速下降。
• 阈值Tm ≈ 0.7对稀疏编码至关重要：低于该值时，仅有少数像素被编码；高于该值时，多个概念被激活，从而实现鲁棒的符号化表征。
• 视觉概念的稀疏二值编码能够有效支持少样本学习和聚类，并简化图像模式的生成建模。