[Paper Review] MelNet: A Generative Model for Audio in the Frequency Domain

Sean Vasquez, Mike Lewis|arXiv (Cornell University)|Jun 4, 2019

Music and Audio Processing50 references111 citations

TL;DR

MelNet generates high-fidelity audio by modeling high-resolution spectrograms with a multiscale, autoregressive two-dimensional time-frequency model, enabling unconditional speech/music generation and end-to-end text-to-speech. It outperforms prior time-domain models in capturing long-range structure and fidelity.

ABSTRACT

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

Motivation & Objective

• Motivate modeling audio in the frequency domain to capture long-range dependencies more tractably than time-domain waveforms.
• Develop an expressive autoregressive model over high-resolution spectrograms to reduce information loss and oversmoothing.
• Propose a multiscale (coarse-to-fine) generation procedure to jointly capture local details and global structure in audio.
• Demonstrate broad applicability to unconditional speech, music generation, and end-to-end text-to-speech.

Proposed method

• Model spectrograms as a product of conditional distributions using a Gaussian mixture model per spectrogram element.
• Parameterize each conditional distribution with a neural network conditioned on the preceding context.
• Use a two-stack autoregressive network: a time-delayed stack (captures information from earlier frames) and a frequency-delayed stack (captures within-frame history and outputs from the time-delayed stack).
• Optionally include a centralized stack and conditioning mechanisms for text/speaker inputs.
• Employ a multiscale generation approach that partitions spectrograms into tiers (x1,…,xG) generated in coarse-to-fine order, with interleaved conditioning across tiers.
• Training uses recursive tier partitioning (split along time or frequency) and tier-specific networks; sampling interleaves generated tiers to form full spectrograms.
• End-to-end conditioning for text-to-speech via learned alignment (location-based attention) between spectrogram frames and character sequences.

Experimental results

Research questions

• RQ1 Can modeling high-resolution spectrograms with a fully autoregressive, 2D time-frequency model capture long-range audio structure beyond what time-domain models achieve?
• RQ2 Does multiscale (coarse-to-fine) generation improve fidelity and reduce oversmoothing relative to single-scale autoregressive spectrogram models?
• RQ3 Is MelNet broadly applicable to unconditional speech, music, and end-to-end text-to-speech generation, and how does it perform versus waveform-based baselines in density estimation and human judgments?
• RQ4 Can end-to-end TTS be achieved by learning alignment between spectrograms and text within this framework?

Key findings

• MelNet learns to generate coherent long-range structure in speech and music better than a WaveNet baseline in human judgments on long samples.
• On unconditional tasks, MelNet produces samples with consistent prosody and speech characteristics across single-speaker, multi-speaker, and piano music data.
• In end-to-end TTS settings, MelNet can perform multi-speaker synthesis and capture multimodal vocal renditions conditioned on text and speaker inputs.
• Qualitative results show that MelNet can maintain speaker characteristics across primed sequences and generate variations in speaking style and noise conditions within VoxCeleb2 data.
• The multiscale generation procedure enables high-resolution spectrograms to be produced with preserved global structure and detailed local fidelity.