[论文解读] Multi-Grained Spatio-temporal Modeling for Lip-reading

Chenhao Wang|arXiv (Cornell University)|Aug 30, 2019

Speech and Audio Processing参考文献 26被引用 40

一句话总结

本文提出一个两分支前端（2D ResNet-34 细粒度特征和 3D-DenseNet 中粒度特征）具有可学习的空间注意力融合，随后使用两层双向 ConvLSTM，并加入前向输入注意力，在词级唇读任务的 LRW 与 LRW-1000 上实现了最先进的结果。

ABSTRACT

Lip-reading aims to recognize speech content from videos via visual analysis of speakers' lip movements. This is a challenging task due to the existence of homophemes-words which involve identical or highly similar lip movements, as well as diverse lip appearances and motion patterns among the speakers. To address these challenges, we propose a novel lip-reading model which captures not only the nuance between words but also styles of different speakers, by a multi-grained spatio-temporal modeling of the speaking process. Specifically, we first extract both frame-level fine-grained features and short-term medium-grained features by the visual front-end, which are then combined to obtain discriminative representations for words with similar phonemes. Next, a bidirectional ConvLSTM augmented with temporal attention aggregates spatio-temporal information in the entire input sequence, which is expected to be able to capture the coarse-gained patterns of each word and robust to various conditions in speaker identity, lighting conditions, and so on. By making full use of the information from different levels in a unified framework, the model is not only able to distinguish words with similar pronunciations, but also becomes robust to appearance changes. We evaluate our method on two challenging word-level lip-reading benchmarks and show the effectiveness of the proposed method, which also demonstrate the above claims.

研究动机与目标

• 在同源音素和说话者/风格变异下激发稳健的唇读能力。
• 捕捉逐帧的细粒度特征与短期中粒度动力学。
• 通过可学习的空间注意机制融合多粒度特征。

提出的方法

• 使用一个 2D ResNet-34 分支提取逐帧的细粒度特征。
• 使用一个 3D-DenseNet（52 层）分支捕捉中粒度的短期时空模式。
• 通过一个 1x1x1 卷积学习的适应性空间注意掩码来融合分支。
• 用带前向输入注意力的两层双向 ConvLSTM 处理融合后的特征。
• 在对各分支进行预训练后进行端到端训练。
• 在 LRW 和 LRW-1000 词级唇读基准上进行评估。

实验结果

研究问题

• RQ1 将细粒度的二维逐帧特征与中粒度的三维时空特征结合，是否能提升词级唇读的性能？
• RQ2 可学习的空间注意力融合是否比简单拼接更好地整合多粒度特征？
• RQ3 带前向输入注意力的 ConvLSTM 是否提升跨说话者与条件的长期时空建模？

主要发现

• 在 LRW 上，提出的模型达到 83.34% 的准确率，超越了若干基线。
• 在 LRW-1000 上，提出的模型达到 36.91% 的准确率。
• 两分支结合 Bi-ConvLSTM 的表现优于两分支结合 Bi-GRU，且前向输入注意力带来进一步提升。
• 通过可学习的空间注意掩码进行融合，比简单融合获得更好的表示。
• 带前向输入注意力的 Bi-ConvLSTM 能更好地处理帧重要性与长距离依赖。
• 与复现的基线相比，该方法在各数据集上呈现持续的改进。