[Paper Review] Multi-Grained Spatio-temporal Modeling for Lip-reading

Chenhao Wang|arXiv (Cornell University)|Aug 30, 2019

Speech and Audio Processing26 references40 citations

TL;DR

The paper proposes a two-branch front-end (2D ResNet-34 fine-grained and 3D-DenseNet medium-grained) with a learnable spatial attention fusion, followed by a two-layer Bi-ConvLSTM with forward input attention, achieving state-of-the-art results on LRW and LRW-1000 for word-level lip-reading.

ABSTRACT

Lip-reading aims to recognize speech content from videos via visual analysis of speakers' lip movements. This is a challenging task due to the existence of homophemes-words which involve identical or highly similar lip movements, as well as diverse lip appearances and motion patterns among the speakers. To address these challenges, we propose a novel lip-reading model which captures not only the nuance between words but also styles of different speakers, by a multi-grained spatio-temporal modeling of the speaking process. Specifically, we first extract both frame-level fine-grained features and short-term medium-grained features by the visual front-end, which are then combined to obtain discriminative representations for words with similar phonemes. Next, a bidirectional ConvLSTM augmented with temporal attention aggregates spatio-temporal information in the entire input sequence, which is expected to be able to capture the coarse-gained patterns of each word and robust to various conditions in speaker identity, lighting conditions, and so on. By making full use of the information from different levels in a unified framework, the model is not only able to distinguish words with similar pronunciations, but also becomes robust to appearance changes. We evaluate our method on two challenging word-level lip-reading benchmarks and show the effectiveness of the proposed method, which also demonstrate the above claims.

Motivation & Objective

• Motivate robust lip-reading under homophemes and speaker/style variation.
• Capture both fine-grained per-frame features and medium-grained short-term dynamics.
• Fuse multi-granularity features with a learnable spatial attention mechanism.

Proposed method

• Use a 2D ResNet-34 branch to extract fine-grained per-frame features.
• Use a 3D-DenseNet (52-layer) branch to capture medium-grained short-term spatio-temporal patterns.
• Fuse branches via an adaptive spatial attention mask learned by a 1x1x1 convolution.
• Process fused features with a two-layer bidirectional ConvLSTM augmented with forward input attention.
• Train end-to-end after pretraining individual branches.
• Evaluate on LRW and LRW-1000 word-level lip-reading benchmarks.

Experimental results

Research questions

• RQ1 Can combining fine-grained 2D frame features and medium-grained 3D spatio-temporal features improve word-level lip-reading performance?
• RQ2 Does a learnable spatial attention fusion better integrate multi-granularity features than simple concatenation?
• RQ3 Does ConvLSTM with forward input attention improve long-term spatio-temporal modeling for lip-reading across speakers and conditions?

Key findings

• On LRW, the proposed model achieves 83.34% accuracy, outperforming several baselines.
• On LRW-1000, the proposed model achieves 36.91% accuracy.
• Two-branch with Bi-ConvLSTM outperforms two-branch with Bi-GRU, and the forward input attention provides further gains.
• Fusion via a learnable spatial attention mask yields better representations than simple fusion.
• The Bi-ConvLSTM with forward input attention better handles frame importance and long-range dependencies.
• Compared to reproduced baselines, the method shows consistent improvements across datasets.