[Paper Review] Affectron: Emotional Speech Synthesis with Affective and Contextually Aligned Nonverbal Vocalizations

Deok-Hyun Cho, Hyung-Seok Oh+2 more|arXiv (Cornell University)|Mar 15, 2026

Emotion and Mood Recognition0 citations

TL;DR

Affectron fine-tunes a verbal-speech pre-trained neural codec language model to generate expressive nonverbal vocalizations (NVs) aligned with affect and context, using emotion-driven NV matching, emotion-aware NV routing, NV masking, and NV-augmented training on a small open corpus.

ABSTRACT

Nonverbal vocalizations (NVs), such as laughter and sighs, are central to the expression of affective cues in emotional speech synthesis. However, learning diverse and contextually aligned NVs remains challenging in open settings due to limited NV data and the lack of explicit supervision. Motivated by this challenge, we propose Affectron as a framework for affective and contextually aligned NV generation. Built on a small-scale open and decoupled corpus, Affectron introduces an NV-augmented training strategy that expands the distribution of NV types and insertion locations. We further incorporate NV structural masking into a speech backbone pre-trained on purely verbal speech to enable diverse and natural NV synthesis. Experimental results demonstrate that Affectron produces more expressive and diverse NVs than baseline systems while preserving the naturalness of the verbal speech stream.

Motivation & Objective

• Address data scarcity and bias in generating contextually appropriate nonverbal vocalizations (NVs) for emotional TTS.
• Leverage a small-scale open corpus to augment NV diversity without explicit alignment annotations.
• Develop training-time NV augmentation and NV-structural masking to enable affect-aware NV synthesis.
• Fine-tune a verbal-speech pre-trained neural codec language model to jointly model NVs with verbal speech.

Proposed method

• Use a small open, decoupled corpus where verbal speech and NVs are recorded separately to augment NV types and insertion locations.
• Introduce emotion-driven top-K NV matching to select NVs emotionally aligned with each utterance.
• Introduce emotion-aware top-K routing to place selected NVs at contextually appropriate locations based on affective dynamics.
• Incorporate NV structural masking into the speech backbone to condition generation on surrounding verbal affective context.
• Fine-tune a VoiceCraft/Neural Codec Language Model (NCLM) with rearranged NV-augmented token sequences and NV-tagged transcripts.
• During inference, generate NVs conditioned on an NV-tagged transcript and an emotional reference utterance without alignment or routing steps.

Experimental results

Research questions

• RQ1 Can NV types and locations be realistically expanded and aligned to affective context using small open corpora?
• RQ2 Do emotion-driven NV matching and emotion-aware routing improve NV expressiveness and contextual coherence compared to baseline systems?
• RQ3 Does NV structural masking improve naturalness and integration of NVs with verbal speech during generation?
• RQ4 Is NV-augmented training more effective than direct training on verbal and NV data separately for zero-shot and seen-speaker scenarios?

Key findings

• Affectron yields more expressive and diverse NV synthesis while preserving verbal naturalness compared to baseline NCLM-based TTS models.
• Emotion-driven NV matching and emotion-aware routing produce NV-type and location distributions that align more closely with empirical data than competing approaches.
• NV structural masking enhances boundary naturalness and coherence by conditioning generation on surrounding verbal affective context.
• Ablation studies show each component (NV masking, routing, matching, and augmentation) contributes to improved NV realism and diversity.
• Zero-shot and unseen-speaker evaluations indicate Affectron maintains NV quality and contextual alignment across speakers.