[Paper Review] AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization

Yen-Hao Chen, Da-Yi Wu+2 more|arXiv (Cornell University)|Oct 31, 2020

Speech Recognition and Synthesis25 references4 citations

TL;DR

AGAIN-VC proposes a one-shot voice conversion system using a single encoder with activation guidance and adaptive instance normalization to disentangle speaker and content representations. By applying a learnable sigmoid activation as a soft information bottleneck on content embeddings, it achieves state-of-the-art performance in both objective and subjective evaluations, outperforming models with dual encoders while reducing model size by 30%.

ABSTRACT

Recently, voice conversion (VC) has been widely studied. Many VC systems use disentangle-based learning techniques to separate the speaker and the linguistic content information from a speech signal. Subsequently, they convert the voice by changing the speaker information to that of the target speaker. To prevent the speaker information from leaking into the content embeddings, previous works either reduce the dimension or quantize the content embedding as a strong information bottleneck. These mechanisms somehow hurt the synthesis quality. In this work, we propose AGAIN-VC, an innovative VC system using Activation Guidance and Adaptive Instance Normalization. AGAIN-VC is an auto-encoder-based model, comprising of a single encoder and a decoder. With a proper activation as an information bottleneck on content embeddings, the trade-off between the synthesis quality and the speaker similarity of the converted speech is improved drastically. This one-shot VC system obtains the best performance regardless of the subjective or objective evaluations.

Motivation & Objective

• To address the trade-off between synthesis quality and speaker content disentanglement in one-shot voice conversion.
• To eliminate the need for a separate speaker encoder by reusing channel-wise statistics from instance normalization layers.
• To introduce activation guidance as a soft, learnable information bottleneck to prevent speaker information leakage into content embeddings.
• To improve model efficiency and performance through a unified, single-encoder architecture.
• To achieve state-of-the-art results in both objective metrics and subjective human evaluation.

Proposed method

• A single encoder extracts both content and speaker representations using instance normalization (IN) layers, where channel-wise mean μ and standard deviation σ serve as speaker embeddings.
• Adaptive instance normalization (AdaIN) transfers speaker style from target embeddings to content features during decoding.
• Activation guidance applies a learnable sigmoid function with hyperparameter α to the content embedding, acting as a continuous, soft information bottleneck to suppress speaker-related information.
• The model is trained with a reconstruction loss: ℒ = ||X − X̂||₁, minimizing the difference between input and output mel-spectrograms.
• The activation function is optimized during training, with α = 0.1 found to yield the best balance between disentanglement and reconstruction.
• The architecture avoids separate speaker and content encoders, reducing model size while maintaining or improving performance.

Experimental results

Research questions

• RQ1 Can a single encoder effectively disentangle speaker and content representations in one-shot voice conversion?
• RQ2 Does activation guidance with a learnable sigmoid function improve disentanglement without degrading synthesis quality?
• RQ3 How does the proposed method compare to dual-encoder baselines (e.g., AdaIN-VC) in terms of performance and model efficiency?
• RQ4 What is the optimal hyperparameter α for the activation guidance function in balancing speaker similarity and content fidelity?
• RQ5 Does the proposed method achieve superior results in both objective metrics and subjective human evaluation?

Key findings

• The proposed 1-Enc-sig model achieved the lowest reconstruction error (0.151) among all configurations, outperforming both 1-Enc and 2-Enc baselines.
• Content embedding speaker classification accuracy dropped to 1.7% with the proposed activation guidance, indicating near-perfect disentanglement of speaker information.
• The model achieved 93.2% speaker classification accuracy on speaker embeddings, confirming effective speaker representation learning.
• Subjective evaluations showed the proposed model scored highest in both naturalness (MOS) and speaker similarity, significantly outperforming 1-Enc and 2-Enc baselines.
• The single-encoder model reduced model size by 30% (9.5M parameters) compared to the dual-encoder version (13.5M parameters), without sacrificing performance.
• Sigmoid with α = 0.1 was found to be optimal, balancing information filtering and content preservation, outperforming ReLU, ELU, and Tanh.