[Paper Review] MixSpeech: Cross-Modality Self-Learning with Audio-Visual Stream Mixup for Visual Speech Translation and Recognition

Xize Cheng, Linjun Li+8 more|arXiv (Cornell University)|Mar 9, 2023

Subtitles and Audiovisual Media8 citations

TL;DR

Proposes MixSpeech, a cross-modality self-learning framework that uses mixed audio-visual speech to regularize visual speech translation and lip reading, achieving state-of-the-art results on AVMuST-TED and LRS/LRS2/CMLR datasets.

ABSTRACT

Multi-media communications facilitate global interaction among people. However, despite researchers exploring cross-lingual translation techniques such as machine translation and audio speech translation to overcome language barriers, there is still a shortage of cross-lingual studies on visual speech. This lack of research is mainly due to the absence of datasets containing visual speech and translated text pairs. In this paper, we present extbf{AVMuST-TED}, the first dataset for extbf{A}udio- extbf{V}isual extbf{Mu}ltilingual extbf{S}peech extbf{T}ranslation, derived from extbf{TED} talks. Nonetheless, visual speech is not as distinguishable as audio speech, making it difficult to develop a mapping from source speech phonemes to the target language text. To address this issue, we propose MixSpeech, a cross-modality self-learning framework that utilizes audio speech to regularize the training of visual speech tasks. To further minimize the cross-modality gap and its impact on knowledge transfer, we suggest adopting mixed speech, which is created by interpolating audio and visual streams, along with a curriculum learning strategy to adjust the mixing ratio as needed. MixSpeech enhances speech translation in noisy environments, improving BLEU scores for four languages on AVMuST-TED by +1.4 to +4.2. Moreover, it achieves state-of-the-art performance in lip reading on CMLR (11.1\%), LRS2 (25.5\%), and LRS3 (28.0\%).

Motivation & Objective

• Motivate cross-lingual visual speech research due to scarcity of visual speech with translations.
• Introduce AVMuST-TED, the first audio-visual multilingual speech translation dataset for four languages.
• Develop a cross-modality self-learning framework that regularizes visual speech translation using high-discrimination audio speech.
• Bridge modality gaps with mixed speech to improve knowledge transfer and robustness in noisy environments.
• Demonstrate state-of-the-art lip translation and lip reading across multiple datasets.
• Provide insights into cross-lingual visual speech translation and potential applications.

Proposed method

• Pretrain a translation decoder on high-discrimination audio speech to learn inter-lingual mappings from source phonemes to target language text.
• Align visual speech with audio speech via cross-modality self-learning to transfer the audio-derived mappings to visual speech.
• Synthesize mixed speech by interpolating audio and visual speech at frame level to bridge modality gaps (MixSpeech).
• Use curriculum-learning to adapt the mixing ratio during training based on prediction uncertainty.
• Regularize visual and mixed-speech translations with Jensen-Shannon Divergence to align their output distributions while maintaining audio-vision knowledge via a mixed-objective loss.

Figure 1: Diagram of speech tasks. Audio speech and visual speech are paired parallel speech streams which can be employed for speech recognition and speech translation. However, only Lip-Translation remains unexplored.

Experimental results

Research questions

• RQ1 Can audio speech pretraining improve visual speech translation and reduce cross-modality transfer gaps?
• RQ2 Does interpolating audio and visual streams (mixed speech) further reduce modality gap and boost visual speech translation performance?
• RQ3 Can a curriculum-based mixing strategy adaptively optimize cross-modality knowledge transfer during training?
• RQ4 How does MixSpeech perform on AVMuST-TED lip translation and on standard lip-reading benchmarks (LRS2, LRS3, CMLR) under varying resource conditions?

Key findings

• MixSpeech improves visual speech translation BLEU scores by +1.4 to +4.2 across four languages on AVMuST-TED compared to baselines.
• End-to-end MixSpeech achieves state-of-the-art lip translation on AVMuST-TED and lip reading on LRS2 (25.5%), LRS3 (28.0%), and CMLR (11.1%).
• MixSpeech bridges cross-modality gaps by using mixed speech, with notable gains when mixing ratio is appropriately configured (e.g., En-Es).
• A curriculum-learning-based mixing ratio adapts during training, further boosting translation performance.
• Across noisy audio conditions, MixSpeech maintains robustness and outperforms audio-only baselines in visual speech translation.

Figure 2: Illustration of our proposed MixSpeech. We first pretrain the model with audio speech translation as shown in the dashed boxed, and then train the visual speech translation with mixed speech regularization. The blank dashed boxes denote the modality missing speech.