[论文解读] Deep Convolutional Neural Network Features and the Original Image

Connor J. Parde, Carlos D. Castillo+5 more|arXiv (Cornell University)|Nov 6, 2016

Face recognition and analysis参考文献 17被引用 19

一句话总结

本文研究了深度卷积神经网络（DCNN）特征在对姿态和光照变化具有鲁棒性的情况下，如何保留原始图像属性的信息。通过在线性分类器和t-SNE可视化分析两个面部识别网络，研究发现高层特征编码了偏航角、俯仰角、媒体类型（静态图像与视频）以及图像质量信息——低质量图像在特征空间中靠近原点聚集，表明可提出一种新型方法，用于在DCNN流水线中自动识别低质量输入。

ABSTRACT

Face recognition algorithms based on deep convolutional neural networks (DCNNs) have made progress on the task of recognizing faces in unconstrained viewing conditions. These networks operate with compact feature-based face representations derived from learning a very large number of face images. While the learned features produced by DCNNs can be highly robust to changes in viewpoint, illumination, and appearance, little is known about the nature of the face code that emerges at the top level of such networks. We analyzed the DCNN features produced by two face recognition algorithms. In the first set of experiments we used the top-level features from the DCNNs as input into linear classifiers aimed at predicting metadata about the images. The results show that the DCNN features contain surprisingly accurate information about the yaw and pitch of a face, and about whether the face came from a still image or a video frame. In the second set of experiments, we measured the extent to which individual DCNN features operated in a view-dependent or view-invariant manner. We found that view-dependent coding was a characteristic of the identities rather than the DCNN features - with some identities coded consistently in a view-dependent way and others in a view-independent way. In our third analysis, we visualized the DCNN feature space for over 24,000 images of 500 identities. Images in the center of the space were uniformly of low quality (e.g., extreme views, face occlusion, low resolution). Image quality increased monotonically as a function of distance from the origin. This result suggests that image quality information is available in the DCNN features, such that consistently average feature values reflect coding failures that reliably indicate poor or unusable images. Combined, the results offer insight into the coding mechanisms that support robust representation of faces in DCNNs.

研究动机与目标

• 理解深度卷积神经网络（DCNN）在非约束条件下学习到的面部表征的性质。
• 研究鲁棒的DCNN特征是否保留了原始图像属性（如姿态、光照和媒体类型）的信息。
• 检验视图相关编码是身份的属性，还是DCNN特征本身的属性。
• 探索图像质量是否编码在DCNN特征空间的结构中，特别是与原点距离的关系。

提出的方法

• 在Janus CS2数据集（IJB-A的扩展版本）上训练了两个近期的DCNN面部识别模型。
• 使用这些网络的高层特征作为线性分类器的输入，以预测元数据（偏航角、俯仰角、媒体类型）。
• 应用t分布随机邻域嵌入（t-SNE）技术，可视化包含500个身份的24,502张图像的特征空间。
• 根据图像在特征空间中距离原点的距离对图像进行排序，以评估图像质量趋势。
• 分析个体特征在不同身份之间对视图依赖性和媒体依赖性的表现。
• 可视化靠近原点的低质量图像聚类，以识别常见的退化因素（如遮挡、模糊、极端视角）。

实验结果

研究问题

• RQ1 高层DCNN特征在多大程度上保留了原始图像姿态（偏航角、俯仰角）和媒体类型（静态图像或视频帧）的信息？
• RQ2 视图相关编码是DCNN特征的属性，还是特定身份的属性？
• RQ3 DCNN特征空间中距离原点的距离是否与图像质量相关？
• RQ4 DCNN特征空间的结构能否用于识别并过滤低质量或不可用的图像？

主要发现

• DCNN高层特征保留了关于偏航角和俯仰角的高精度信息，使得线性分类器能够以优异性能预测这些属性。
• 相同特征还能准确预测输入是静态图像还是视频帧，表明编码了媒体特异性特征。
• 视图相关编码被发现是特定身份的属性，而非DCNN特征的普遍特性：某些身份始终以视图相关方式编码，而另一些身份则以视图无关方式编码。
• 低质量图像（表现为极端视角、遮挡、模糊或低分辨率）在特征空间中聚集在原点附近。
• 图像质量随距离原点的距离单调增加，表明距离原点的距离可作为图像质量的可靠代理指标。
• 特征空间结构表明，持续偏低的特征值反映了编码失败，从而可实现对低质量或不可用图像的自动检测。