[Paper Review] Exploring speaker enrolment for few-shot personalisation in emotional vocalisation prediction

Andreas Triantafyllopoulos, Meishu Song+3 more|arXiv (Cornell University)|Jun 14, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes a few-shot personalization framework for emotional vocalization prediction using a novel 'enrolment' encoder that adapts the emotion encoder via dot-product attention using two unlabelled utterances from the target speaker. The method achieves a 2.5% relative improvement, reaching a CCC of 0.650 on the ExVo Few-Shot dev set, with the enrolment encoder contributing most of the gain over the baseline CNN14 model.

ABSTRACT

In this work, we explore a novel few-shot personalisation architecture for emotional vocalisation prediction. The core contribution is an `enrolment' encoder which utilises two unlabelled samples of the target speaker to adjust the output of the emotion encoder; the adjustment is based on dot-product attention, thus effectively functioning as a form of `soft' feature selection. The emotion and enrolment encoders are based on two standard audio architectures: CNN14 and CNN10. The two encoders are further guided to forget or learn auxiliary emotion and/or speaker information. Our best approach achieves a CCC of $.650$ on the ExVo Few-Shot dev set, a $2.5\%$ increase over our baseline CNN14 CCC of $.634$.

Motivation & Objective

• To improve few-shot personalization in emotional vocalization prediction using minimal speaker-specific data.
• To investigate how two unlabelled utterances from a target speaker can be used to adapt emotion recognition models.
• To explore the effectiveness of auxiliary losses in guiding the enrolment and emotion encoders to learn speaker-specific and emotion-invariant representations, respectively.
• To evaluate whether attention-based conditioning via an enrolment encoder outperforms standard adversarial speaker-invariant baselines.

Proposed method

• The model uses a dual-branch architecture: an emotion encoder (CNN14) and an enrolment encoder (CNN10), with the latter conditioning the former via dot-product attention.
• The enrolment encoder processes two unlabelled target speaker utterances to generate a context vector that modulates the emotion encoder’s output through soft attention.
• Auxiliary adversarial heads are applied to both encoders: one to suppress speaker identity in the emotion encoder, and another to enhance speaker representation in the enrolment encoder.
• Gradient reversal layers are used with a warm-up schedule to gradually introduce domain-invariant training, improving robustness.
• The final prediction is made by a shared emotion classifier that takes the attended, speaker-adapted embedding as input.
• Model training is optimized using a combination of emotion regression loss and auxiliary adversarial losses, with hyperparameter tuning for gradient reversal.

Experimental results

Research questions

• RQ1 Can two unlabelled utterances from a target speaker be effectively used to personalize an emotion recognition model in a few-shot setting?
• RQ2 Does attention-based conditioning via an enrolment encoder outperform standard adversarial speaker-invariance techniques in few-shot personalization?
• RQ3 What is the contribution of auxiliary adversarial heads in improving the generalization and robustness of the personalization mechanism?
• RQ4 How do different combinations of auxiliary losses affect performance on the ExVo Few-Shot benchmark?

Key findings

• The proposed enrolment-based personalization method achieves a CCC of 0.650 on the ExVo Few-Shot dev set, representing a 2.5% relative improvement over the baseline CNN14 model (0.634).
• The highest performance is achieved by the architecture with only the enrolment encoder and no additional auxiliary heads, indicating that the core mechanism provides the primary gain.
• Adding an adversarial speaker classifier to the emotion encoder branch significantly degrades performance, suggesting that speaker invariance may not be beneficial in this context.
• The enrolment encoder with an adversarial emotion classifier (but no speaker classifier) yields a CCC of 0.647, showing marginal improvement over the baseline.
• The addition of both adversarial heads to the enrolment encoder results in a CCC of 0.642 on the test set, indicating that auxiliary guidance offers limited benefit beyond the enrolment mechanism itself.
• The ablation study confirms that the enrolment encoder is the primary driver of performance gain, with auxiliary components providing only marginal improvements.