[Paper Review] Recurrent Neural Network Transducer for Audio-Visual Speech Recognition

Takaki Makino, Hank Liao+5 more|arXiv (Cornell University)|Nov 8, 2019

Speech and Audio Processing30 references15 citations

TL;DR

This paper proposes an RNN-T-based audio-visual speech recognition system that leverages a 31k-hour YouTube-derived audio-visual dataset to achieve state-of-the-art performance. By synchronizing audio and visual features and using joint modeling via RNN-T, the system reduces word error rate to 4.5% on the LRS3-TED benchmark and shows significant robustness under noisy or overlapping speech conditions.

ABSTRACT

This work presents a large-scale audio-visual speech recognition system based on a recurrent neural network transducer (RNN-T) architecture. To support the development of such a system, we built a large audio-visual (A/V) dataset of segmented utterances extracted from YouTube public videos, leading to 31k hours of audio-visual training content. The performance of an audio-only, visual-only, and audio-visual system are compared on two large-vocabulary test sets: a set of utterance segments from public YouTube videos called YTDEV18 and the publicly available LRS3-TED set. To highlight the contribution of the visual modality, we also evaluated the performance of our system on the YTDEV18 set artificially corrupted with background noise and overlapping speech. To the best of our knowledge, our system significantly improves the state-of-the-art on the LRS3-TED set.

Motivation & Objective

• To develop a scalable, end-to-end audio-visual speech recognition system that improves robustness in challenging acoustic environments.
• To address the limitations of unimodal ASR systems, particularly in noisy or overlapping speech scenarios.
• To explore the benefits of combining audio and visual modalities using a unified RNN-T architecture with synchronized features.
• To build a large-scale, fully automated audio-visual training dataset from public YouTube videos using face tracking and caption alignment.
• To demonstrate that visual modality significantly enhances recognition performance, especially under degraded audio conditions.

Proposed method

• Automatically mined a 150k-hour audio-only dataset from YouTube using user-uploaded captions and face tracking to extract 31k hours of audio-visual utterances.
• Applied face landmark smoothing and synchronized audio and visual features by aligning audio frame rate to video frame rate using a fixed short-term Fourier transform window.
• Used a VGG-inspired CNN to extract visual embeddings from mouth-region thumbnails, and mel-filterbank coefficients for audio features.
• Employed an RNN-T architecture with separate encoder and decoder networks, where the encoder processes audio and visual inputs, and the decoder generates character-level outputs.
• Jointly trained the encoder, decoder, and joint network using a sequence-to-sequence loss, enabling implicit language modeling and monotonic alignment.
• Enabled end-to-end training with character-level outputs to support open-vocabulary recognition without phoneme-based transcription.

Experimental results

Research questions

• RQ1 Can an RNN-T-based audio-visual ASR system achieve state-of-the-art performance on large-vocabulary benchmarks like LRS3-TED?
• RQ2 How does the visual modality improve recognition robustness in the presence of background noise or overlapping speech?
• RQ3 To what extent does a large-scale, automatically constructed audio-visual dataset enhance system performance compared to smaller, curated datasets?
• RQ4 How does the performance of visual-only ASR vary across different video quality conditions, such as face size, pose, and resolution?
• RQ5 Does the RNN-T architecture outperform CTC or attention-based models in audio-visual speech recognition under real-world conditions?

Key findings

• The audio-visual RNN-T system achieved a word error rate (WER) of 4.5% on the LRS3-TED benchmark, representing a state-of-the-art result at the time of publication.
• The visual-only system achieved a WER of 33.6% on LRS3-TED, significantly outperforming prior visual-only models like CTC-V2P (55.1%) and demonstrating strong performance on high-quality studio recordings.
• Under artificial noise corruption, the audio-visual system maintained a WER of 4.5%, showing substantial improvement over the audio-only system (4.8%) and robustness in degraded conditions.
• On the more challenging YTDev18 set, the visual-only system had a WER of 48.5%, dropping to 57.0% on low-quality video segments, highlighting sensitivity to image quality and real-world variability.
• The audio-visual system outperformed the TM-seq2seq model (7.2% WER) on the LRS3-TED set, demonstrating the advantage of using a larger dataset and the RNN-T architecture.
• The system achieved a 4.8% WER on audio-only training and inference, and 4.5% on audio-visual, showing that visual input provides a consistent and measurable improvement even with large-scale audio-only data.