[Paper Review] On Robustness to Missing Video for Audiovisual Speech Recognition

Oscar Chang, Otavio Braga+3 more|arXiv (Cornell University)|Dec 13, 2023

Speech and Audio Processing4 citations

TL;DR

This paper proposes a robust audiovisual speech recognition framework that ensures performance degradation does not occur when video is missing. It introduces a standardized evaluation protocol and demonstrates that a cascade-based architecture-agnostic method consistently outperforms existing techniques like dropout in maintaining accuracy across varying video loss conditions.

ABSTRACT

It has been shown that learning audiovisual features can lead to improved speech recognition performance over audio-only features, especially for noisy speech. However, in many common applications, the visual features are partially or entirely missing, e.g.~the speaker might move off screen. Multi-modal models need to be robust: missing video frames should not degrade the performance of an audiovisual model to be worse than that of a single-modality audio-only model. While there have been many attempts at building robust models, there is little consensus on how robustness should be evaluated. To address this, we introduce a framework that allows claims about robustness to be evaluated in a precise and testable way. We also conduct a systematic empirical study of the robustness of common audiovisual speech recognition architectures on a range of acoustic noise conditions and test suites. Finally, we show that an architecture-agnostic solution based on cascades can consistently achieve robustness to missing video, even in settings where existing techniques for robustness like dropout fall short.

Motivation & Objective

• To address the lack of consensus in evaluating robustness to missing video in audiovisual speech recognition (AVASR).
• To establish a precise, testable framework for evaluating robustness in AVASR models.
• To empirically assess the robustness of common AVASR architectures under diverse noise and video dropout conditions.
• To develop and validate an architecture-agnostic solution that ensures video absence never degrades performance below audio-only baselines.

Proposed method

• Proposes a formal robustness criterion: a model is robust if additional video input at test time can only improve or maintain performance, never hurt it.
• Introduces three distinct test suites—$ au_{\text{mid}}$, $ au_{\text{end}}$, and $ au_{\text{rate}}$—to simulate missing video at different temporal positions and rates.
• Employs a cascade-based architecture where a separate audio-only model is fused with a video-augmented model, enabling robust inference regardless of video availability.
• Uses WER (Word Error Rate) as the primary metric across multiple noise conditions (clean, 0dB, 10dB, 20dB SNR) and video dropout levels.
• Applies standard AVASR models (e.g., Conformer, LSTM) with and without video, comparing performance under AV, AO, and AO baseline conditions.
• Validates robustness across diverse architectures and training regimes, ensuring generalizability of the cascade solution.

Figure 1: a) An audiovisual model trained with the RNN-T loss. b) A cascaded audiovisual model. The dashed lines indicate different routes the input can take: the AO path, which goes through the audio model, and the AV path, which goes through the audiovisual model.

Experimental results

Research questions

• RQ1 Can a standardized, testable framework be defined to evaluate robustness to missing video in AVASR models?
• RQ2 How do existing AVASR models perform under controlled video dropout across different noise levels and dropout patterns?
• RQ3 Does the cascade-based architecture-agnostic solution consistently achieve robustness across diverse model architectures and noise conditions?
• RQ4 Is there a discrepancy between claimed robustness and actual performance when video is missing, as revealed by comprehensive evaluation?

Key findings

• The proposed robustness framework enables precise, reproducible evaluation of AVASR models under missing video conditions, resolving ambiguity in prior work.
• Existing models like those from Makino et al. (2019) and Zhang et al. (2019) fail robustness criteria, showing degraded performance when tested on audio-only data despite being trained on audiovisual data.
• The cascade-based method achieves consistent robustness across all tested settings: WER remains stable or improves as video is progressively removed, even under high noise (0dB SNR).
• For the LSTM-based cascade model, WER on AO baseline remains at 24.54±1.45 WER, while performance under 100% video dropout is 25.04±1.5 WER—indicating no degradation.
• The Conformer-based cascade model maintains WER below 4.5 WER across all noise levels and dropout rates, demonstrating strong generalization and robustness.
• Even in extreme cases (e.g., 100% video dropout), the cascade model’s performance is not worse than the audio-only baseline, validating the core robustness intuition.

Figure 2: Dropping $50\%$ of the frames under different test suites.