[Paper Review] End-to-end Audio-visual Speech Recognition with Conformers

Pingchuan Ma, Stavros Petridis+1 more|arXiv (Cornell University)|Feb 12, 2021

Speech and Audio Processing36 references4 citations

TL;DR

This paper presents an end-to-end audio-visual speech recognition model using Conformers for joint audio and visual feature extraction from raw waveforms and pixels, with a hybrid CTC/attention decoder and a transformer-based language model. The model achieves state-of-the-art performance on LRS2 and LRS3, reducing word error rates to 1.2% (audio-visual) on LRS3 v0.0 and 2.3% on LRS3 v0.4, outperforming prior methods despite using significantly less training data.

ABSTRACT

In this work, we present a hybrid CTC/Attention model based on a ResNet-18 and Convolution-augmented transformer (Conformer), that can be trained in an end-to-end manner. In particular, the audio and visual encoders learn to extract features directly from raw pixels and audio waveforms, respectively, which are then fed to conformers and then fusion takes place via a Multi-Layer Perceptron (MLP). The model learns to recognise characters using a combination of CTC and an attention mechanism. We show that end-to-end training, instead of using pre-computed visual features which is common in the literature, the use of a conformer, instead of a recurrent network, and the use of a transformer-based language model, significantly improve the performance of our model. We present results on the largest publicly available datasets for sentence-level speech recognition, Lip Reading Sentences 2 (LRS2) and Lip Reading Sentences 3 (LRS3), respectively. The results show that our proposed models raise the state-of-the-art performance by a large margin in audio-only, visual-only, and audio-visual experiments.

Motivation & Objective

• To develop an end-to-end audio-visual speech recognition model that directly processes raw audio and video inputs without pre-computed features.
• To improve performance by replacing recurrent networks with Conformers and using a transformer-based language model.
• To demonstrate that joint end-to-end training on raw inputs yields superior results compared to models using pre-extracted features.
• To evaluate the model on large-scale, in-the-wild datasets LRS2 and LRS3, including both clean and noisy conditions.
• To show that audio-visual fusion significantly improves robustness in high-noise environments.

Proposed method

• The model uses a modified ResNet-18 with 3D convolutions for visual input and 1D convolutions for audio input to extract raw features from pixels and waveforms.
• Visual and audio features are processed by a Conformer encoder, which combines convolutional and self-attention mechanisms for temporal modeling.
• Features from both modalities are fused via a Multi-Layer Perceptron (MLP) before being passed to a hybrid CTC/attention decoder.
• The model is trained end-to-end using CTC and attention loss, with a transformer-based language model for improved decoding.
• Raw audio waveforms are used instead of log-Mel filter-bank features to enhance robustness in noisy conditions.
• The architecture is evaluated on LRS2 and LRS3, with ablation studies on data size, input modality, and training strategy.

Fig. 1 : End-to-end audio-visual speech recognition architecture. The inputs are pixels and raw audio waveforms.

Experimental results

Research questions

• RQ1 Can end-to-end training on raw audio and visual inputs improve performance in audio-visual speech recognition compared to models using pre-computed features?
• RQ2 Does replacing recurrent networks with Conformers lead to significant performance gains in audio-visual speech recognition?
• RQ3 How does using a transformer-based language model affect recognition accuracy compared to RNN-based models?
• RQ4 Can a model trained on a smaller dataset outperform those trained on much larger datasets, especially in noisy or challenging conditions?
• RQ5 How does audio-visual fusion enhance robustness in high-noise environments?

Key findings

• The proposed model achieves a word error rate (WER) of 1.2% on LRS3 v0.0 for audio-visual recognition, setting a new state-of-the-art.
• On LRS3 v0.4, the model achieves a WER of 2.3% for audio-visual recognition, outperforming prior methods by 2.2 percentage points.
• The visual-only model achieves a WER of 43.3% on LRS3 v0.4, outperforming previous methods despite using less training data.
• The audio-only model achieves a WER of 1.3% on LRS3 v0.0, reducing to 2.3% on v0.4, demonstrating robustness of raw audio training.
• The model outperforms a method trained on 31,000 hours of data (YT) with only 438 hours of LRS3 data, showing data efficiency.
• Audio-visual fusion significantly improves performance, especially in high-noise conditions, confirming the complementary nature of modalities.

Fig. 2 : Word Error Rate (WER) as a function of the noise level. A: End-to-End audio model. V: End-to-End visual model, AV: End-to-End audio-visual model. log-Mel filter-bank: A conformer model trained with log-Mel filter-bank features.