[论文解读] Robust LSTM-Autoencoders for Face De-Occlusion in the Wild

Fang Zhao, Jiashi Feng+3 more|arXiv (Cornell University)|Dec 27, 2016

Face recognition and analysis参考文献 19被引用 6

一句话总结

本文提出了一种用于真实世界非约束环境下人脸去遮挡的鲁棒LSTM自编码器（RLA）模型。通过使用多尺度空间LSTM编码器和双通道LSTM解码器，该模型逐步联合重建人脸并检测遮挡，有效恢复被遮挡的人脸，同时通过身份保持的对抗性训练保留身份判别特征，在合成数据集和真实世界遮挡数据集上均实现了最先进的恢复质量与识别准确率。

ABSTRACT

Face recognition techniques have been developed significantly in recent years. However, recognizing faces with partial occlusion is still challenging for existing face recognizers which is heavily desired in real-world applications concerning surveillance and security. Although much research effort has been devoted to developing face de-occlusion methods, most of them can only work well under constrained conditions, such as all the faces are from a pre-defined closed set. In this paper, we propose a robust LSTM-Autoencoders (RLA) model to effectively restore partially occluded faces even in the wild. The RLA model consists of two LSTM components, which aims at occlusion-robust face encoding and recurrent occlusion removal respectively. The first one, named multi-scale spatial LSTM encoder, reads facial patches of various scales sequentially to output a latent representation, and occlusion-robustness is achieved owing to the fact that the influence of occlusion is only upon some of the patches. Receiving the representation learned by the encoder, the LSTM decoder with a dual channel architecture reconstructs the overall face and detects occlusion simultaneously, and by feat of LSTM, the decoder breaks down the task of face de-occlusion into restoring the occluded part step by step. Moreover, to minimize identify information loss and guarantee face recognition accuracy over recovered faces, we introduce an identity-preserving adversarial training scheme to further improve RLA. Extensive experiments on both synthetic and real datasets of faces with occlusion clearly demonstrate the effectiveness of our proposed RLA in removing different types of facial occlusion at various locations. The proposed method also provides significantly larger performance gain than other de-occlusion methods in promoting recognition performance over partially-occluded faces.

研究动机与目标

• 解决在遮挡类型和位置多变的真实世界非约束场景中的人脸去遮挡挑战，且测试样本来自训练过程中未见过的开放集。
• 克服现有去遮挡方法依赖封闭集假设或在复杂、空间连续遮挡（如太阳镜或口罩）下失效的局限性。
• 开发一种渐进式、端到端可训练的框架，通过循环建模与联合重建-检测机制，逐步恢复被遮挡的面部区域。
• 在恢复的人脸中保留判别性身份特征，以提升下游人脸识别性能，尤其是在严重遮挡情况下。
• 提出一种身份保持的对抗性训练方案，以减少伪影并提升视觉保真度，同时保持识别准确率。

提出的方法

• 采用多尺度空间LSTM编码器，按顺序处理不同尺寸的面部块，生成对遮挡具有鲁棒性的潜在表征，最小化被遮挡块对编码的影响。
• 实现双通道LSTM解码器，包含两个子网络：一个用于人脸重建，另一个用于遮挡检测，二者通过隐藏状态反馈实现交叉注意力协作。
• 利用循环解码逐步从粗到细地恢复被遮挡的面部区域，利用先前恢复的信息引导后续步骤。
• 使用重建图像与真实图像之间的均方误差（MSE）损失，端到端训练自编码器，以确保像素级保真度。
• 集成一个监督CNN，通过对齐恢复人脸与原始人脸的特征表示，强制在恢复过程中保持身份一致性。
• 应用对抗判别器以区分真实与恢复的无遮挡人脸，减少伪影并提升感知质量，同时保持身份信息。

Figure 1: We address the task of face de-occlusion with various types of occlusions under the condition of open test sets (i.e., test samples have no identical subject with training samples). (a) Original occlusion-free faces; (b) Occluded faces; (c) Recovered faces by our proposed method.

实验结果

研究问题

• RQ1 循环自编码器架构是否能有效恢复真实世界非约束环境下多种遮挡类型的人脸？
• RQ2 基于LSTM的编码与解码实现渐进式、分步去遮挡，是否在视觉质量与识别准确率上优于单步去遮挡方法？
• RQ3 在双通道LSTM框架中联合进行人脸重建与遮挡检测，是否能提升被遮挡面部区域的定位与恢复效果？
• RQ4 身份保持的对抗性训练在多大程度上提升了恢复人脸的判别性质量，以支持下游识别任务？
• RQ5 所提出的RLA模型在开放集测试场景下的泛化能力如何，即测试样本未出现在训练集中？

主要发现

• 在带有遮挡的人脸LFW数据集上，所提出的RLA模型实现了10.0%的最低等错误率（EER），显著优于PCA、AE、SRC和SSDA等基线方法。
• 在50OccPeople数据集上，RLA实现了17.3%的平均EER，表现出对真实世界遮挡数据的强大泛化能力，并优于所有对比方法。
• IP-RLA变体（引入身份保持监督）相比标准自编码器（AE）将EER降低了12.4个百分点，相比PCA降低了18.5个百分点，显著提升了识别准确率。
• 带有遮挡检测的双通道LSTM解码器相比仅人脸重建，将错误率降低了最多10.5个百分点，证明遮挡定位有助于特征保持。
• 对抗性训练显著减少了恢复人脸中的视觉伪影，同时保持了高身份判别性，这一点通过提升的识别性能与感知质量得到验证。
• 模型在半脸遮挡（如左半或右半脸）情况下表现最佳，面部对称性有助于恢复，但在上半脸与下半脸遮挡情况下仍表现出色，表明跨区域特征迁移有效。

Figure 2: Illustration on the framework of our proposed robust LSTM-Autoencoders and its training process. It consists of a multi-scale spatial LSTM encoder and a dual-channel LSTM decoder for concurrent face reconstruction and occlusion detection.