[Paper Review] Learning Speech Representations from Raw Audio by Joint Audiovisual Self-Supervision

Abhinav Shukla, Stavros Petridis+1 more|arXiv (Cornell University)|Jul 8, 2020

Speech and Audio Processing22 references4 citations

TL;DR

This paper proposes a joint audiovisual self-supervision method to learn speech representations directly from raw audio waveforms by combining audio-only self-supervision (predicting audio attributes) with visual self-supervision (generating talking faces from audio). The approach significantly outperforms fully supervised training and existing self-supervised methods, especially under low-data and noisy conditions, demonstrating the effectiveness of multimodal pretraining for speech representation learning.

ABSTRACT

The intuitive interaction between the audio and visual modalities is valuable for cross-modal self-supervised learning. This concept has been demonstrated for generic audiovisual tasks like video action recognition and acoustic scene classification. However, self-supervision remains under-explored for audiovisual speech. We propose a method to learn self-supervised speech representations from the raw audio waveform. We train a raw audio encoder by combining audio-only self-supervision (by predicting informative audio attributes) with visual self-supervision (by generating talking faces from audio). The visual pretext task drives the audio representations to capture information related to lip movements. This enriches the audio encoder with visual information and the encoder can be used for evaluation without the visual modality. Our method attains competitive performance with respect to existing self-supervised audio features on established isolated word classification benchmarks, and significantly outperforms other methods at learning from fewer labels. Notably, our method also outperforms fully supervised training, thus providing a strong initialization for speech related tasks. Our results demonstrate the potential of multimodal self-supervision in audiovisual speech for learning good audio representations.

Motivation & Objective

• To develop a self-supervised method that learns robust speech representations from raw audio waveforms using cross-modal supervision.
• To explore the utility of visual self-supervision—specifically, generating talking faces from audio—as a pretext task to enrich audio representations.
• To evaluate whether joint audiovisual self-supervision improves performance on downstream speech classification tasks, especially when labeled data is scarce.
• To compare the proposed method against audio-only self-supervised methods and fully supervised training, particularly in low-resource and noisy settings.

Proposed method

• The method employs a 1D ResNet18 as the audio encoder, trained via joint self-supervision from raw audio and visual modalities.
• Audio-only self-supervision is achieved by training decoders to reconstruct salient audio attributes (MFCCs, log mel-spectrograms, and raw waveforms) from the audio encoder's latent representations.
• Visual self-supervision is implemented through a video generation head that reconstructs a sequence of talking face frames from the first frame and the audio embedding, enforcing alignment between audio and lip movements.
• The audio encoder is jointly optimized using reconstruction losses from both audio and visual tasks, enabling it to learn representations enriched with visual information.
• The final audio encoder is fine-tuned on downstream classification tasks using a BiGRU classifier with cross-entropy loss.
• The method is evaluated on isolated word classification (SPC and LRW datasets), with ablation studies on limited data (10%) and noisy conditions (SNR from -5 dB to 20 dB).

Experimental results

Research questions

• RQ1 Can joint audiovisual self-supervision improve speech representation learning from raw audio compared to unimodal audio-only self-supervision?
• RQ2 Does visual self-supervision via talking face generation enhance the generalization of audio representations, especially in low-data regimes?
• RQ3 Can the proposed method outperform fully supervised training when fine-tuned on downstream speech classification tasks?
• RQ4 How does the method perform under noisy conditions, and does it generalize better than baselines in low-SNR environments?

Key findings

• On the full SPC dataset, the proposed method achieves 95.21% accuracy, ranking second among self-supervised methods and outperforming fully supervised training.
• On the LRW dataset with full data, the method achieves 95.37% accuracy, the best among all compared methods.
• With only 10% of training data, the method achieves 90.63% accuracy on SPC and 77.13% on LRW—significantly outperforming other self-supervised and fully supervised baselines.
• In noisy conditions (SNR from -5 dB to 20 dB), the joint audiovisual method consistently outperforms MFCCs and all other self-supervised baselines across both SPC and LRW datasets.
• The model trained with joint audiovisual self-supervision generalizes better than audio-only or visual-only baselines, demonstrating the complementary value of multimodal supervision.
• The method significantly outperforms fully supervised training from scratch, indicating that self-supervised pretraining with cross-modal signals is a powerful initialization for speech tasks.