[Paper Review] A Cascade Sequence-to-Sequence Model for Chinese Mandarin Lip Reading

Ya Zhao, Rui Xu+1 more|arXiv (Cornell University)|Aug 14, 2019

Speech and Audio Processing13 references4 citations

TL;DR

This paper proposes a cascade sequence-to-sequence model (CSSMCM) for Chinese Mandarin lip reading that explicitly models lexical tones using visual and syntactic cues, improving sentence-level recognition. By jointly predicting pinyin, tone, and character sequences with multi-modal attention, CSSMCM achieves state-of-the-art performance on the newly collected CMLR dataset, demonstrating that tone modeling significantly reduces character error rates in Mandarin lip reading.

ABSTRACT

Lip reading aims at decoding texts from the movement of a speaker's mouth. In recent years, lip reading methods have made great progress for English, at both word-level and sentence-level. Unlike English, however, Chinese Mandarin is a tone-based language and relies on pitches to distinguish lexical or grammatical meaning, which significantly increases the ambiguity for the lip reading task. In this paper, we propose a Cascade Sequence-to-Sequence Model for Chinese Mandarin (CSSMCM) lip reading, which explicitly models tones when predicting sentence. Tones are modeled based on visual information and syntactic structure, and are used to predict sentence along with visual information and syntactic structure. In order to evaluate CSSMCM, a dataset called CMLR (Chinese Mandarin Lip Reading) is collected and released, consisting of over 100,000 natural sentences from China Network Television website. When trained on CMLR dataset, the proposed CSSMCM surpasses the performance of state-of-the-art lip reading frameworks, which confirms the effectiveness of explicit modeling of tones for Chinese Mandarin lip reading.

Motivation & Objective

• To address the high ambiguity in Chinese Mandarin lip reading caused by tone distinctions, which are absent in English.
• To develop a multimodal lip reading framework that jointly models visual cues, pinyin, and tone for improved sentence-level recognition.
• To collect and release a large-scale, naturalistic Chinese Mandarin lip reading dataset (CMLR) for research and evaluation.
• To demonstrate that explicit tone modeling enhances character-level accuracy in Mandarin lip reading.
• To outperform existing end-to-end and two-stage lip reading models on sentence-level recognition tasks.

Proposed method

• A three-stage cascade architecture: first predicting pinyin sequences from video, then predicting tone sequences using video and pinyin inputs, and finally predicting character sequences using all three inputs.
• Dual attention mechanism in the tone prediction sub-network to attend to both video frames and pinyin sequences simultaneously.
• Triplet attention mechanism in the final sub-network to integrate visual, pinyin, and tone representations for character generation.
• End-to-end joint fine-tuning of all three sub-networks to optimize overall performance.
• Use of syllables (not individual pinyin characters) as the pinyin unit to reduce decoding complexity and improve modeling stability.
• Leveraging neck and head movements as visual cues for tone prediction, based on prior linguistic findings linking tone production to visible articulatory features.

Experimental results

Research questions

• RQ1 Can explicit modeling of lexical tones improve sentence-level lip reading performance in tone-based languages like Mandarin?
• RQ2 How does the integration of visual, pinyin, and tone information affect character sequence prediction accuracy?
• RQ3 Does using syllables instead of individual pinyin characters as input units improve the robustness and accuracy of sequence-to-sequence lip reading models?
• RQ4 How do attention mechanisms in the cascade model align video frames with predicted pinyin, tones, and characters?
• RQ5 To what extent do failure cases in the model stem from homophones or similar-sounding characters with different tones?

Key findings

• CSSMCM achieves a lower character error rate than state-of-the-art models, confirming the effectiveness of explicit tone modeling in Mandarin lip reading.
• The model successfully predicts homophones with different tones—such as '实惠' (affordable) versus '事会' (opportunity)—where other models fail due to tone misclassification.
• Attention visualization shows that the model focuses on relevant video frames and uses future tone information as a language model to refine character predictions.
• The model outperforms both WAS and LipCH-Net, especially in cases involving tone distinctions with identical mouth shapes.
• Failure cases are primarily due to homophones or characters with identical finals, highlighting the challenge of tone ambiguity even with advanced modeling.
• The CMLR dataset, containing over 100,000 natural sentences from news broadcasts, enables reliable evaluation and future benchmarking.