[Paper Review] Vesper: A Compact and Effective Pretrained Model for Speech Emotion Recognition

Weidong Chen, Xiaofen Xing+2 more|arXiv (Cornell University)|Jul 20, 2023

Emotion and Mood RecognitionPsychology3 citations

TL;DR

This paper proposes Vesper, a compact and effective pretrained model for speech emotion recognition that improves upon WavLM through emotion-guided masking and hierarchical self-supervision. Despite having only 4 layers, Vesper outperforms WavLM Base (12 layers), and Vesper-12 surpasses WavLM Large (24 layers) on IEMOCAP, MELD, and CREMA-D datasets, demonstrating superior performance with reduced model size.

ABSTRACT

This paper presents a paradigm that adapts general large-scale pretrained models (PTMs) to speech emotion recognition task. Although PTMs shed new light on artificial general intelligence, they are constructed with general tasks in mind, and thus, their efficacy for specific tasks can be further improved. Additionally, employing PTMs in practical applications can be challenging due to their considerable size. Above limitations spawn another research direction, namely, optimizing large-scale PTMs for specific tasks to generate task-specific PTMs that are both compact and effective. In this paper, we focus on the speech emotion recognition task and propose an improved emotion-specific pretrained encoder called Vesper. Vesper is pretrained on a speech dataset based on WavLM and takes into account emotional characteristics. To enhance sensitivity to emotional information, Vesper employs an emotion-guided masking strategy to identify the regions that need masking. Subsequently, Vesper employs hierarchical and cross-layer self-supervision to improve its ability to capture acoustic and semantic representations, both of which are crucial for emotion recognition. Experimental results on the IEMOCAP, MELD, and CREMA-D datasets demonstrate that Vesper with 4 layers outperforms WavLM Base with 12 layers, and the performance of Vesper with 12 layers surpasses that of WavLM Large with 24 layers.

Motivation & Objective

• To address the limitations of general-purpose pretrained models (PTMs) in speech emotion recognition, which are often too large and not optimized for emotional content.
• To develop a task-specific PTM that is both compact and effective by combining model compression and label-free adaptation.
• To enhance sensitivity to emotional cues through a novel emotion-guided masking strategy during pretraining.
• To improve representation learning by integrating hierarchical and cross-layer self-supervision for better capture of acoustic and semantic features.
• To validate that the last-layer representation of Vesper is sufficient for emotion recognition, simplifying downstream deployment.

Proposed method

• Vesper is initialized with pre-trained WavLM weights to leverage general speech representations.
• An emotion-guided masking strategy identifies high-energy speech regions likely to contain emotional content, focusing masking on these areas to improve emotional sensitivity.
• Hierarchical self-supervision is applied across layers to strengthen learning of acoustic features in shallow layers and semantic features in deeper layers.
• Cross-layer self-supervision enhances feature refinement by encouraging consistency between representations across different depths of the network.
• The model is fine-tuned on downstream speech emotion recognition tasks using standard classification heads without requiring labeled data during pretraining.
• Model compression is achieved by reducing the number of layers (e.g., 4 or 12 instead of 12 or 24), resulting in a smaller, more efficient architecture.

Experimental results

Research questions

• RQ1 Can a task-specific, compact pretrained model outperform larger, general-purpose models in speech emotion recognition?
• RQ2 Does an emotion-guided masking strategy improve the model's sensitivity to emotional content during self-supervised pretraining?
• RQ3 Can hierarchical and cross-layer self-supervision enhance the learning of both acoustic and semantic representations critical for emotion recognition?
• RQ4 Is the last-layer representation of Vesper sufficient for high-performance emotion recognition, enabling simplified deployment?
• RQ5 To what extent does Vesper reduce computational cost while maintaining or improving accuracy compared to WavLM variants?

Key findings

• Vesper with 4 layers achieves better performance than WavLM Base (12 layers) across all three datasets: IEMOCAP, MELD, and CREMA-D.
• Vesper-12 outperforms WavLM Large (24 layers) on all evaluation metrics, demonstrating that a smaller model can surpass a larger one when specifically optimized for emotion recognition.
• On IEMOCAP, Vesper-4 achieves a 6.2% to 6.6% improvement in F1-score over WavLM Base when using all layers, and a 1.4% to 1.9% improvement when using only the last layer.
• For Vesper-12, using only the last-layer representation yields a 0.2% to 0.4% performance increase compared to the full SUPERB setting, indicating the last-layer representation is highly informative.
• t-SNE visualization confirms that Vesper-12 successfully clusters representations of four emotions (happy, sad, angry, neutral), with minimal overlap, unlike WavLM models.
• The ablation study confirms that both emotion-guided masking and the hierarchical/cross-layer self-supervision components contribute significantly to performance gains.