[Paper Review] RAVE: A variational autoencoder for fast and high-quality neural audio synthesis

Antoine Caillon, Philippe Esling|arXiv (Cornell University)|Nov 9, 2021

Speech and Audio ProcessingComputer Science40 citations

TL;DR

RAVE introduces a two-stage VAE with adversarial fine-tuning and multiband waveform decomposition to achieve 48kHz high-quality audio synthesis at about 20x real-time on CPU.

ABSTRACT

Deep generative models applied to audio have improved by a large margin the state-of-the-art in many speech and music related tasks. However, as raw waveform modelling remains an inherently difficult task, audio generative models are either computationally intensive, rely on low sampling rates, are complicated to control or restrict the nature of possible signals. Among those models, Variational AutoEncoders (VAE) give control over the generation by exposing latent variables, although they usually suffer from low synthesis quality. In this paper, we introduce a Realtime Audio Variational autoEncoder (RAVE) allowing both fast and high-quality audio waveform synthesis. We introduce a novel two-stage training procedure, namely representation learning and adversarial fine-tuning. We show that using a post-training analysis of the latent space allows a direct control between the reconstruction fidelity and the representation compactness. By leveraging a multi-band decomposition of the raw waveform, we show that our model is the first able to generate 48kHz audio signals, while simultaneously running 20 times faster than real-time on a standard laptop CPU. We evaluate synthesis quality using both quantitative and qualitative subjective experiments and show the superiority of our approach compared to existing models. Finally, we present applications of our model for timbre transfer and signal compression. All of our source code and audio examples are publicly available.

Motivation & Objective

• Motivate fast, high-quality neural audio synthesis without heavy autoregressive generation.
• Develop a VAE-based framework that balances reconstruction fidelity with latent space compactness.
• Enable 48kHz audio synthesis with low computational cost using multiband waveform decomposition.
• Provide a post-training latent space analysis method to identify informative latent dimensions.
• Demonstrate applications in timbre transfer and signal compression.

Proposed method

• Propose a two-stage training procedure: first train a regular VAE for representation learning, then fine-tune with an adversarial generation objective.
• Use a multiband decomposition of the raw waveform to reduce temporal dimensionality and enable 48kHz synthesis.
• Optimize the encoder with a multiscale spectral loss during representation learning (stage 1).
• Freeze the encoder in stage 2 and train the decoder with a hinge-GAN objective plus spectral and feature-matching losses.
• Perform post-training latent-space analysis via a singular value decomposition to separate informative and uninformative latent dimensions, enabling a variable fidelity reconstruction.

Experimental results

Research questions

• RQ1 Can a VAE-based model achieve high-quality 48kHz audio synthesis with real-time or near-real-time performance on CPU?
• RQ2 How can the latent space be analyzed and pruned post-training to balance reconstruction fidelity and representation compactness?
• RQ3 Does adversarial fine-tuning after representation learning improve perceptual quality without sacrificing the learned latent structure?
• RQ4 Can multiband waveform decomposition enable high sampling-rate synthesis with manageable computational cost?
• RQ5 Is the model capable of timbre transfer and signal compression without supervised targets?

Key findings

• RAVE achieves 48kHz audio synthesis with 20x real-time speed on a standard laptop CPU.
• In a 15-trial MOS study on strings data, RAVE scores 3.01 vs 2.68 (NSynth) and 1.15 (SING).
• RAVE uses 17.6M parameters, substantially fewer than the baselines.
• A 16-band multiband decomposition enables high-quality 48kHz synthesis with lower computational load.
• Post-training latent-space analysis via SVD yields a fidelity parameter f that reduces the latent dimensionality significantly while controlling reconstruction quality.
• RAVE supports timbre transfer and latent-space-based signal compression.