[Paper Review] Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu+2 more|arXiv (Cornell University)|Jan 5, 2022

Speech and Audio Processing113 citations

TL;DR

AV-HuBERT learns self-supervised audio-visual speech representations by masking multimodal inputs and predicting iteratively refined multimodal cluster targets, achieving state-of-the-art lip-reading and ASR with substantially less labeled data.

ABSTRACT

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert

Motivation & Objective

• Motivate learning speech representations from correlated audio and lip-visual cues in video data.
• Develop a self-supervised framework that fuses audio and visual streams to learn robust, transferable speech representations.
• Reduce reliance on large labeled corpora for lip-reading while improving downstream ASR performance.

Proposed method

• Introduce AV-HuBERT, a multimodal encoder that masks both audio and visual streams and predicts cluster assignments.
• Use modality-specific encoders plus a shared transformer backbone to fuse modalities.
• Iteratively refine target clusters by clustering features (e.g., MFCC) and using learned AV features to re-cluster.
• Apply a masking strategy that substitutes visual frames with imposter segments to create a harder prediction task.
• Incorporate modality dropout to prevent audio-dominant solutions and bridge pre-training with single-modality fine-tuning.
• Fine-tune with CTC or S2S losses; evaluate lip-reading and ASR performance; explore self-training with pseudo-labels.

Experimental results

Research questions

• RQ1 Can a self-supervised, multimodal speech model improve lip-reading when trained with limited labeled data?
• RQ2 Do multimodal targets (audio-visual) yield more transferable representations than unimodal targets for lip-reading and ASR?
• RQ3 What is the impact of modality dropout and the proposed masking strategy on learning robust AV speech representations?
• RQ4 How does AV-HuBERT perform in multilingual vs. monolingual pre-training settings and in ASR downstream tasks?

Key findings

• AV-HuBERT achieves 32.5% WER on LRS3 with only 30 hours of labeled data, outperforming a 33.6% WER model trained on ~31K hours of transcribed data.
• Using all 433 hours of LRS3 for pre-training and fine-tuning reduces lip-reading WER to 28.6%.
• Combining AV-HuBERT with self-training yields a new lip-reading SOTA of 26.9% WER with 433 hours and 30 hours of labeled data.
• For ASR, pre-training an audio-HuBERT with targets from AV-HuBERT achieves 1.3% WER (S2S) on LRS3, surpassing prior state-of-the-art without external LM.
• AV-HuBERT pre-training (multimodal targets) improves lip-reading and ASR compared to unimodal visual or audio targets, and targets driven by audio-visual features yield the best results.
• The approach remains effective when pre-training data is multilingual, though language-domain alignment influences gains; domain-matched pre-training data is advantageous.