[Paper Review] High Fidelity Neural Audio Compression

Alexandre Défossez, Jade Copet+2 more|arXiv (Cornell University)|Oct 24, 2022

Speech and Audio ProcessingComputer Science279 citations

TL;DR

EnCodec is a real-time neural audio codec that achieves high-fidelity compression across 24 kHz mono and 48 kHz stereo, using end-to-end training with RVQ, perceptual discriminators, and optional Transformer-based entropy coding to reach state-of-the-art MUSHRA scores at low bitrates.

ABSTRACT

We introduce a state-of-the-art real-time, high-fidelity, audio codec leveraging neural networks. It consists in a streaming encoder-decoder architecture with quantized latent space trained in an end-to-end fashion. We simplify and speed-up the training by using a single multiscale spectrogram adversary that efficiently reduces artifacts and produce high-quality samples. We introduce a novel loss balancer mechanism to stabilize training: the weight of a loss now defines the fraction of the overall gradient it should represent, thus decoupling the choice of this hyper-parameter from the typical scale of the loss. Finally, we study how lightweight Transformer models can be used to further compress the obtained representation by up to 40%, while staying faster than real time. We provide a detailed description of the key design choices of the proposed model including: training objective, architectural changes and a study of various perceptual loss functions. We present an extensive subjective evaluation (MUSHRA tests) together with an ablation study for a range of bandwidths and audio domains, including speech, noisy-reverberant speech, and music. Our approach is superior to the baselines methods across all evaluated settings, considering both 24 kHz monophonic and 48 kHz stereophonic audio. Code and models are available at github.com/facebookresearch/encodec.

Motivation & Objective

• Motivate the need for high-fidelity, low-bitrate neural audio compression in streaming contexts.
• Propose an end-to-end streaming codec with residual vector quantization and perceptual losses to minimize artifacts.
• Stabilize training with a novel loss balancer and evaluate impact of lightweight Transformer-based entropy coding.

Proposed method

• Encoder-decoder convolutional architecture producing a latent z.
• Residual Vector Quantization (RVQ) of encoder output with multiple codebooks.
• A small Transformer language model for entropy coding over discrete RVQ codes.
• Multi-scale STFT-based discriminators (MS-STFT) for perceptual loss.
• Time-domain and frequency-domain reconstruction losses plus adversarial and RVQ commitment losses.
• A loss balancer that scales gradients to keep training stable across heterogeneous losses.

Experimental results

Research questions

• RQ1 Can a streaming neural audio codec provide real-time, high-fidelity audio at low bitrates across speech and music domains?
• RQ2 What architectural and training choices (RVQ, discriminators, and entropy coding) yield state-of-the-art perceptual quality at 24–48 kHz?
• RQ3 How does a gradient balancer affect training stability and interpretability of loss weights?
• RQ4 What bandwidths and sampling rates can be supported with acceptable subjective quality (MUSHRA)?

Key findings

• EnCodec achieves state-of-the-art MUSHRA scores across 24 kHz mono and 48 kHz stereo at multiple bitrates (1.5–12 kbps 24 kHz; 6–24 kbps 48 kHz).
• Transformer-based entropy coding can reduce bandwidth by 20–40% without perceptual degradation for certain use cases.
• MS-STFT-based perceptual loss with RVQ and streaming-friendly padding yields high-quality samples and stable training.
• Streaming (1-pass, CPU-real-time) encoding/decoding is feasible with initial latency around 13 ms at 24 kHz and real-time processing for 6 kbps, with higher latency at 48 kHz.
• Ablation shows MS-STFT discriminator suffices for perceptual quality; additional MPD yields marginal gains.
• Stereo evaluation shows EnCodec competitive with MP3 at 64 kbps and superior to Opus at 6 kbps for music.