[Paper Review] Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning

Chen Chen, Yu‐Chen Hu+4 more|arXiv (Cornell University)|Dec 10, 2022

Speech and Audio Processing4 citations

TL;DR

This paper proposes MSRL, a reinforcement learning-based framework that dynamically integrates modality-invariant and modality-specific visual representations to improve audio-visual speech recognition (AVSR) under noisy conditions. By using a custom reward function aligned with word error rate (WER), MSRL outperforms state-of-the-art methods across clean and diverse noisy settings, including unseen noise types, demonstrating superior robustness and generalization.

ABSTRACT

Audio-visual speech recognition (AVSR) has gained remarkable success for ameliorating the noise-robustness of speech recognition. Mainstream methods focus on fusing audio and visual inputs to obtain modality-invariant representations. However, such representations are prone to over-reliance on audio modality as it is much easier to recognize than video modality in clean conditions. As a result, the AVSR model underestimates the importance of visual stream in face of noise corruption. To this end, we leverage visual modality-specific representations to provide stable complementary information for the AVSR task. Specifically, we propose a reinforcement learning (RL) based framework called MSRL, where the agent dynamically harmonizes modality-invariant and modality-specific representations in the auto-regressive decoding process. We customize a reward function directly related to task-specific metrics (i.e., word error rate), which encourages the MSRL to effectively explore the optimal integration strategy. Experimental results on the LRS3 dataset show that the proposed method achieves state-of-the-art in both clean and various noisy conditions. Furthermore, we demonstrate the better generality of MSRL system than other baselines when test set contains unseen noises.

Motivation & Objective

• Address the over-reliance on audio modality in existing AVSR models, especially under low SNR conditions where visual modality becomes critical.
• Overcome the limitations of fixed fusion strategies that fail to adapt to dynamic, non-stationary noise distributions in real-world scenarios.
• Improve model generalization by effectively utilizing noise-invariant visual modality-specific representations during auto-regressive decoding.
• Develop a flexible integration mechanism that balances modality-invariant and modality-specific representations based on task-specific performance metrics.
• Demonstrate the effectiveness of reinforcement learning in learning optimal fusion policies for AVSR, particularly in low-resource and unseen noise settings.

Proposed method

• Employ a pre-trained vision model to extract modality-specific visual representations from lip movements, independent of the audio stream.
• Introduce a reinforcement learning (RL) agent that dynamically selects between modality-invariant and modality-specific representations during auto-regressive decoding.
• Design a task-specific reward function based on word error rate (WER) to guide the RL agent toward optimal integration strategies.
• Use beam search during inference to generate multiple hypotheses, enabling the RL agent to explore diverse token-level integration policies.
• Train the policy network end-to-end using policy gradient methods, where the agent learns to weigh representations based on noise conditions and context.
• Integrate the RL-based fusion mechanism into an autoregressive ASR decoder, allowing context-aware, adaptive fusion at each decoding step.

Experimental results

Research questions

• RQ1 Can modality-specific visual representations improve AVSR performance when audio is corrupted by noise, especially in low-SNR conditions?
• RQ2 How can a dynamic fusion strategy between modality-invariant and modality-specific representations be learned effectively under varying noise distributions?
• RQ3 Does reinforcement learning with a WER-aligned reward function lead to better generalization on unseen noise types compared to fixed fusion baselines?
• RQ4 To what extent do modality-specific representations enhance data efficiency in low-resource training scenarios?
• RQ5 Can the proposed method outperform state-of-the-art AVSR models in both clean and diverse noisy conditions, including unseen noise types?

Key findings

• MSRL achieves a 1.33% WER in clean conditions on the LRS3 dataset, representing a 5% relative reduction over the best baseline (AV-HuBERT) in normal-resource mode.
• In low-resource mode, MSRL reduces WER to 2.82%, a 14.5% relative improvement over AV-HuBERT, demonstrating superior data efficiency.
• On unseen noise types—cafe, meeting, river, and resto—MSRL outperforms AV-HuBERT by 12.5% to 27.1% relatively in low-resource mode, showing strong generalization.
• In noisy conditions, MSRL surpasses AV-HuBERT by 32.5% (relative) on babble noise and 25.7% on speech noise in low-resource mode, with consistent gains across all SNR levels.
• Even with only 30 hours of labeled data, MSRL outperforms RNN-T, which uses 34,000 hours of data, highlighting its data efficiency.
• The model ensemble baseline underperforms in clean conditions due to over-reliance on visual modality-specific representations, but MSRL avoids this by learning context-aware fusion, maintaining robustness across all settings.