[Paper Review] Unsupervised Audiovisual Synthesis via Exemplar Autoencoders

Kangle Deng, Aayush Bansal+1 more|arXiv (Cornell University)|Jan 13, 2020

Speech and Audio Processing70 references4 citations

TL;DR

This paper proposes Exemplar Autoencoders for unsupervised audiovisual synthesis, enabling zero-shot voice and facial animation transfer from any input speaker to a target speaker using only 3 minutes of target audio-video data. By learning a personalized bottleneck representation that projects out-of-distribution speech onto the target's style while preserving linguistic content, the method achieves state-of-the-art results in both audio and video synthesis without requiring input speaker data or fine-tuning on large datasets.

ABSTRACT

We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. Our approach builds on simple autoencoders that project out-of-sample data onto the distribution of the training set. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target exemplar speech. In contrast to existing methods, the proposed approach can be easily extended to an arbitrarily large number of speakers and styles using only 3 minutes of target audio-video data, without requiring {\em any} training data for the input speaker. To do so, we learn audiovisual bottleneck representations that capture the structured linguistic content of speech. We outperform prior approaches on both audio and video synthesis, and provide extensive qualitative analysis on our project page -- https://www.cs.cmu.edu/~exemplar-ae/.

Motivation & Objective

• To enable zero-shot audiovisual synthesis from any input speaker to a target speaker without requiring training data for the input speaker.
• To learn a personalized audiovisual bottleneck representation that disentangles linguistic content from speaker-specific style and prosody.
• To achieve high-fidelity audio and video synthesis using only 3 minutes of target data, making the method scalable to arbitrarily many speakers.
• To outperform existing zero-shot and supervised approaches in both audio quality and visual realism on in-the-wild web data.
• To provide a plug-and-play, data-efficient solution for applications in healthcare, education, and entertainment.

Proposed method

• The method employs Exemplar Autoencoders that project out-of-sample input speech onto the distribution of a single target speaker’s audiovisual data using a small bottleneck representation.
• The autoencoder is trained end-to-end on 3 minutes of target audio-video data, learning to reconstruct the target’s voice, prosody, and visual appearance while preserving linguistic content.
• The audio bottleneck captures structured phonetic content, enabling content-preserving style transfer from any input speaker to the target.
• Video synthesis is achieved by jointly training a video decoder conditioned on the audio bottleneck, leveraging pre-trained audio features for improved realism.
• The approach uses a WaveNet vocoder for high-fidelity audio reconstruction and avoids reliance on pre-trained speaker embeddings, which may fail to capture in-the-wild stylistic details.
• The method is data-efficient and generalizes to unseen speakers and languages without retraining or paired data.

Experimental results

Research questions

• RQ1 Can a single autoencoder be trained on only 3 minutes of target audio-video data to enable high-quality audiovisual synthesis for any input speaker?
• RQ2 Does projecting input speech onto a target’s distribution via a small bottleneck preserve linguistic content while transferring style?
• RQ3 Can the audio bottleneck learned from a target speaker be effectively transferred to synthesize realistic facial video without additional training data?
• RQ4 How does the performance of Exemplar Autoencoders compare to zero-shot and supervised baselines in terms of voice similarity and content consistency?
• RQ5 Can the method generalize to in-the-wild, uncurated web data and still achieve high-quality synthesis without speaker-specific fine-tuning?

Key findings

• The proposed method achieves a voice similarity score of 99.6% on the Celeb-Audio dataset using WaveNet, outperforming Auto-VC (98.5%) and Chou et al. (57.0%) on speaker classification accuracy.
• With WaveNet, the method achieves a mean cospaced distance (MCD) of 420.3, slightly higher than Auto-VC (408.8), but still yielding high perceptual quality.
• Without WaveNet, the method still outperforms non-WaveNet baselines, achieving 97.0% voice similarity and 420.3 MCD, demonstrating robustness to vocoder choice.
• In video synthesis, finetuning the audio decoder while training the video decoder improves performance, reducing MSE to 76.401 and increasing PSNR to 29.616 on VoxCeleb.
• Ablation studies confirm that using a pre-trained audio model and finetuning the audio bottleneck significantly improves video synthesis quality over training from scratch or training only the video head.
• Human evaluation shows that outputs from the proposed method are preferred 90.6% of the time over a multi-speaker autoencoder, confirming superior style transfer and realism.