[Paper Review] WaveFlow: A Compact Flow-based Model for Raw Audio

Wei Ping, Kainan Peng+2 more|arXiv (Cornell University)|Dec 3, 2019

Music and Audio Processing47 references36 citations

TL;DR

WaveFlow presents a compact flow-based model for raw audio trained with maximum likelihood that achieves high-fidelity speech like WaveNet but with far faster generation and a much smaller footprint.

ABSTRACT

In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$ imes$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$ imes$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.

Motivation & Objective

• Motivate efficient, high-fidelity raw-audio generation with a compact flow-based model.
• Bridge likelihood-based flow models and autoregressive models for 1-D waveform data.
• Demonstrate that a small-footprint WaveFlow can match autoregressive performance with significantly faster synthesis.
• Show that WaveFlow reduces the parameter count relative to WaveGlow while maintaining or improving fidelity.

Proposed method

• Introduce WaveFlow which squeezes 1-D waveforms into a 2-D matrix and models local variations with expressive autoregressive functions.
• Use a dilated 2-D convolutional architecture to implement the conditional shifts and scales in a flow-based mapping.
• Define a triangular Jacobian enabling parallel likelihood computation and autoregressive synthesis across the height dimension.
• Train directly with maximum likelihood without auxiliary losses or density distillation.
• Provide a unified view where WaveNet and WaveGlow are special cases of flow-based models.
• Experiment with various heights h and permutation strategies to balance model capacity and inference parallelism.

Experimental results

Research questions

• RQ1 Can a compact 2-D dilated conv flow model achieve likelihood close to autoregressive WaveNet for raw audio?
• RQ2 How does WaveFlow scale its height parameter h to trade off generation speed and model capacity?
• RQ3 What is the impact of permutation strategies on likelihood and fidelity when stacking multiple Flow layers?
• RQ4 How does WaveFlow compare to WaveGlow and autoregressive Flow in terms of parameters, likelihood, and synthesis speed?
• RQ5 To what extent can likelihood correlate with perceptual speech quality in Flow-based vocoders?

Key findings

• A moderate-sized WaveFlow matches WaveNet in likelihood and produces high-fidelity speech with thousands of time-steps generated in few sequential steps.
• WaveFlow significantly outperforms WaveGlow in likelihood at similar parameter scales (e.g., 5.023 vs. 5.026 with comparable sizes).
• Increasing the squeezed height h improves likelihood; larger models with more flows yield better LLs and MOS while also enabling faster synthesis up to a point.
• WaveFlow with small footprints (5.91M parameters) can synthesize 22.05 kHz speech at 42.60× real-time speed, outperforming WaveGlow in both efficiency and fidelity in several settings.
• Permutation strategies across height dimension (especially mixed reverse and bipartite arrangements) substantially improve likelihood due to bidirectional modeling.
• Text-to-speech experiments show WaveFlow achieves MOS comparable to WaveNet while maintaining a small footprint and fast synthesis.