[论文解读] ACVAE-VC: Non-parallel many-to-many voice conversion with auxiliary classifier variational autoencoder

Hirokazu Kameoka, Takuhiro Kaneko+2 more|arXiv (Cornell University)|Aug 13, 2018

Speech Recognition and Synthesis参考文献 40被引用 48

一句话总结

ACVAE-VC 引入了一种基于带有辅助分类器的 VAE、具全卷积编码器/解码器、信息理论正则化和谱细节移植的非并行多对多语音转换方法，以提升质量和说话人相似度。

ABSTRACT

This paper proposes a non-parallel many-to-many voice conversion (VC) method using a variant of the conditional variational autoencoder (VAE) called an auxiliary classifier VAE (ACVAE). The proposed method has three key features. First, it adopts fully convolutional architectures to construct the encoder and decoder networks so that the networks can learn conversion rules that capture time dependencies in the acoustic feature sequences of source and target speech. Second, it uses an information-theoretic regularization for the model training to ensure that the information in the attribute class label will not be lost in the conversion process. With regular CVAEs, the encoder and decoder are free to ignore the attribute class label input. This can be problematic since in such a situation, the attribute class label will have little effect on controlling the voice characteristics of input speech at test time. Such situations can be avoided by introducing an auxiliary classifier and training the encoder and decoder so that the attribute classes of the decoder outputs are correctly predicted by the classifier. Third, it avoids producing buzzy-sounding speech at test time by simply transplanting the spectral details of the input speech into its converted version. Subjective evaluation experiments revealed that this simple method worked reasonably well in a non-parallel many-to-many speaker identity conversion task.

研究动机与目标

• Motivate a non-parallel, many-to-many VC method that handles unaligned data without transcripts or time alignment.
• Develop a CVAE-based VC framework that preserves attribute information through an auxiliary classifier regularization.
• Improve speech naturalness by transplanting spectral details to avoid oversmoothing.
• Capture temporal dependencies in acoustic features via fully convolutional architectures.
• Provide a practical non-parallel VC solution that outperforms VAE-GAN baselines in subjective tests.

提出的方法

• Use fully convolutional encoder, latent space, and decoder to process acoustic feature sequences rather than frame-by-frame data.
• Introduce an auxiliary classifier with a variational information maximization objective to ensure the attribute label c influences the output.
• Adopt an information-theoretic regularization that maximizes a lower bound on I(c; x|z) via an auxiliary classifier rψ(c|x).
• Avoid oversmoothing by transplanting spectral details from input speech into the converted output rather than relying on Gaussian decoder outputs alone.
• Convert mel-cepstral coefficients derived from WORLD spectral envelopes, using a spectral detail transplantation approach with a WORLD vocoder.
• Implement a fully convolutional gated CNN architecture for encoder, decoder, and auxiliary classifier to handle variable-length sequences.

实验结果

研究问题

• RQ1 Can non-parallel, many-to-many voice conversion be effectively learned without parallel data using an ACVAE framework?
• RQ2 Does incorporating an auxiliary classifier and information-theoretic regularization preserve attribute control (c) in generated speech?
• RQ3 Does transplanting spectral details mitigate oversmoothing and improve perceptual quality compared to VAE-GAN-based approaches?
• RQ4 Do fully convolutional networks capture temporal dependencies in acoustic feature sequences for VC tasks?
• RQ5 How does ACVAE-VC perform in subjective evaluations of sound quality and speaker similarity against baseline methods?

主要发现

• Subjective AB and ABX tests show ACVAE-VC outperforms the VAE-GAN baseline in sound quality and speaker similarity for non-parallel many-to-many VC.
• The method demonstrates improved perceptual quality by transferring spectral details rather than relying on decoders alone.
• ACVAE-VC maintains control over target attributes through the auxiliary classifier regularization, addressing the issue of c being ignored in vanilla CVAEs.
• Fully convolutional architectures enable learning of time dependencies in acoustic feature sequences, improving conversion realism.
• The approach works on a non-parallel corpus with multiple speaker identities and a four-dimension target/source configuration in the reported setup.