[论文解读] Can We Read Speech Beyond the Lips? Rethinking RoI Selection for Deep Visual Speech Recognition

Yuanhang Zhang, Shuang Yang+3 more|arXiv (Cornell University)|Mar 6, 2020

Speech and Audio Processing参考文献 34被引用 6

一句话总结

本文挑战了视觉语音识别（VSR）领域长期聚焦于唇部区域的常规做法，证明唇部以外的面部区域（如面颊和上半部分面部）可显著提升识别性能。通过在对齐的面部输入上应用简单的Cutout增强方法，该方法迫使模型从非唇部区域学习判别性特征，在词级和句级基准测试中均取得最先进结果，相较于仅使用唇部的基线模型，性能提升最高达6.5%。

ABSTRACT

Recent advances in deep learning have heightened interest among researchers in the field of visual speech recognition (VSR). Currently, most existing methods equate VSR with automatic lip reading, which attempts to recognise speech by analysing lip motion. However, human experience and psychological studies suggest that we do not always fix our gaze at each other's lips during a face-to-face conversation, but rather scan the whole face repetitively. This inspires us to revisit a fundamental yet somehow overlooked problem: can VSR models benefit from reading extraoral facial regions, i.e. beyond the lips? In this paper, we perform a comprehensive study to evaluate the effects of different facial regions with state-of-the-art VSR models, including the mouth, the whole face, the upper face, and even the cheeks. Experiments are conducted on both word-level and sentence-level benchmarks with different characteristics. We find that despite the complex variations of the data, incorporating information from extraoral facial regions, even the upper face, consistently benefits VSR performance. Furthermore, we introduce a simple yet effective method based on Cutout to learn more discriminative features for face-based VSR, hoping to maximise the utility of information encoded in different facial regions. Our experiments show obvious improvements over existing state-of-the-art methods that use only the lip region as inputs, a result we believe would probably provide the VSR community with some new and exciting insights.

研究动机与目标

• 探究视觉语音识别（VSR）模型是否能从唇部以外的面部区域获益，挑战长期以来认为仅唇部运动具有信息量的假设。
• 评估不同面部区域（嘴部、整张脸、上半张脸、面颊）在多种数据集和任务中对VSR性能的贡献。
• 开发一种简单而有效的数据增强方法，以促进模型从唇部以外的面部区域学习稳健且具有判别性的特征。
• 减少对显式唇部区域裁剪的依赖，推动一种更整体化、以面部为中心的VSR方法，使其更符合人类视觉扫描行为。

提出的方法

• 作者采用Cutout——一种通过在输入帧中随机遮挡矩形区域的数据增强技术——以促使模型在训练过程中关注非唇部面部区域。
• 通过面部关键点对输入面部进行对齐，以确保空间一致性；感兴趣区域（RoI）被定义为整张脸、嘴部、上半张脸或面颊，用于比较不同区域的性能表现。
• 使用ResNet-34作为主干网络，并通过双向GRU实现时序建模，模型在词级（LRW）和句级（GRID, LRW-1000）基准上进行端到端训练。
• 利用显著性图和时空遮蔽技术解释模型注意力机制，识别对预测至关重要的面部区域。
• 将该方法与基于注意力的RoI选择方法（如CBAM）进行对比，评估隐式注意力机制是否能实现类似性能提升。
• 在不同姿态条件下（偏航旋转）评估性能，以测试模型对头部姿态变化的鲁棒性。

Figure 1: Common practices for RoI selection in VSR. To date, there is no clear consensus on a best practice, resulting in very different RoIs in different works (see Sec. II-A ). Top row: examples of frames from talking face videos. Bottom row: some examples of cropped RoIs in prior work, from left

实验结果

研究问题

• RQ1 视觉语音识别模型是否能从唇部以外的面部区域（如面颊或上半张脸）获益？
• RQ2 通过Cutout进行数据增强是否能通过强制模型关注唇部以外的面部线索，从而提升模型泛化能力？
• RQ3 在词级和句级VSR任务中，不同面部区域对识别性能的贡献如何？
• RQ4 像Cutout这样简单的增强方法是否能优于更复杂的注意力机制，以学习非唇部视觉语音特征？
• RQ5 所提出方法在真实场景中对头部姿态变化的鲁棒性如何？

主要发现

• Cutout增强模型在LRW-1000数据集上达到45.24%的准确率，相较于仅使用唇部的基线模型（38.64%）提升了6.5%。
• 即使仅遮挡7×7的补丁，Cutout模型的性能下降不足2%，而普通模型性能下降高达40%，证明其在唇部以外区域对遮挡具有更强鲁棒性。
• 显著性图显示，Cutout模型学会关注面颊、眉毛和上半张脸，而不仅限于唇部，表明其成功学习了非唇部区域的判别性特征。
• 在Cutout模型中，面对姿态变化时，其性能在简单和中等姿态子集（偏航角≥20°和≥40°）上相较基线提升约2%。
• 时空遮蔽实验确认，面颊和上半张脸等唇部以外区域对预测有显著贡献，当这些区域被遮挡时，性能下降超过10%。
• Cutout增强模型的表现优于CBAM增强模型，表明显式注意力机制在捕捉细微唇部以外线索方面，不如简单数据增强方法有效。

Figure 2: Illustration of the sub-face RoIs defined in this paper. We train baselines on the whole face (blue), the upper face (purple), the cheeks (orange), and the mouth (red).