[Paper Review] Deformation Flow Based Two-Stream Network for Lip Reading

Jingyun Xiao, Shuang Yang+3 more|arXiv (Cornell University)|Mar 12, 2020

Speech and Audio Processing16 references4 citations

TL;DR

This paper proposes a Deformation Flow Based Two-Stream Network (DFTN) for lip reading that leverages self-supervised deformation flow estimation between adjacent video frames to capture motion dynamics in the lip region. By fusing raw video and deformation flow features through a two-stream architecture with bidirectional knowledge distillation, the method achieves state-of-the-art performance on LRW (84.13%) and LRW-1000 (41.93%) benchmarks, demonstrating the effectiveness of complementary motion and appearance cues.

ABSTRACT

Lip reading is the task of recognizing the speech content by analyzing movements in the lip region when people are speaking. Observing on the continuity in adjacent frames in the speaking process, and the consistency of the motion patterns among different speakers when they pronounce the same phoneme, we model the lip movements in the speaking process as a sequence of apparent deformations in the lip region. Specifically, we introduce a Deformation Flow Network (DFN) to learn the deformation flow between adjacent frames, which directly captures the motion information within the lip region. The learned deformation flow is then combined with the original grayscale frames with a two-stream network to perform lip reading. Different from previous two-stream networks, we make the two streams learn from each other in the learning process by introducing a bidirectional knowledge distillation loss to train the two branches jointly. Owing to the complementary cues provided by different branches, the two-stream network shows a substantial improvement over using either single branch. A thorough experimental evaluation on two large-scale lip reading benchmarks is presented with detailed analysis. The results accord with our motivation, and show that our method achieves state-of-the-art or comparable performance on these two challenging datasets.

Motivation & Objective

• To improve lip reading by modeling fine-grained temporal motion dynamics in the lip region using deformation flows between adjacent frames.
• To address the limitation of existing methods that underutilize spatiotemporal correlations in video frames by introducing a motion-aware modality.
• To enhance feature learning by enabling knowledge exchange between two-stream branches via bidirectional knowledge distillation.
• To achieve superior performance on word-level lip reading benchmarks using complementary cues from raw video and deformation flow.

Proposed method

• A self-supervised Deformation Flow Network (DFN) is trained to estimate pixel-wise deformation fields between consecutive video frames, representing motion as apparent deformations in the lip region.
• The deformation flow and raw grayscale frames are fed into separate streams of a two-stream convolutional network for feature extraction.
• Bidirectional knowledge distillation is applied during training, where each stream distills knowledge from the other’s soft probability predictions to improve generalization and feature learning.
• At inference, predictions from both streams are fused via element-wise multiplication of probability distributions to improve classification accuracy.
• The method uses ResNet-based backbones and cross-entropy loss with intermediate fusion and knowledge distillation for joint optimization.
• The deformation flow is computed using a flow estimation module trained without any human-annotated motion labels, relying solely on video consistency.

Experimental results

Research questions

• RQ1 Can deformation flow between adjacent frames effectively model lip motion dynamics for improved lip reading?
• RQ2 Does combining raw video and deformation flow features through a two-stream network yield better performance than single-stream baselines?
• RQ3 Can bidirectional knowledge distillation between two-stream branches further enhance model accuracy and generalization?
• RQ4 How does the proposed method compare to state-of-the-art lip reading models on large-scale benchmarks?

Key findings

• The proposed DFTN achieves a word-level recognition accuracy of 84.13% on the LRW benchmark, surpassing the previous state-of-the-art method (84.07%).
• On the more challenging LRW-1000 dataset, DFTN achieves 41.93% accuracy, significantly outperforming prior methods such as Yang19 (38.19%) and Wang19 (36.91%).
• The bidirectional knowledge distillation loss improves the performance of both individual streams and the final fused prediction, demonstrating effective inter-stream knowledge transfer.
• Multiplicative fusion of probability outputs from the two streams yields higher accuracy than additive fusion or intermediate feature fusion, indicating complementary information.
• The self-supervised DFN generates meaningful deformation flows without any labeled motion data, proving effective for motion modeling in lip reading.
• Ablation studies confirm that the combination of deformation flow and raw video, along with bidirectional distillation, is key to the performance gain.