[Paper Review] Streamable Neural Audio Synthesis With Non-Causal Convolutions

Antoine Caillon, Philippe Esling|arXiv (Cornell University)|Apr 14, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes a post-training method to make non-causal convolutional neural networks streamable in real-time audio processing by introducing internal delays, enabling high-quality, low-latency audio synthesis without retraining. The approach transforms existing models like RAVE into real-time compatible systems with no quality loss and superior speed over overlap-add methods.

ABSTRACT

Deep learning models are mostly used in an offline inference fashion. However, this strongly limits the use of these models inside audio generation setups, as most creative workflows are based on real-time digital signal processing. Although approaches based on recurrent networks can be naturally adapted to this buffer-based computation, the use of convolutions still poses some serious challenges. To tackle this issue, the use of causal streaming convolutions have been proposed. However, this requires specific complexified training and can impact the resulting audio quality. In this paper, we introduce a new method allowing to produce non-causal streaming models. This allows to make any convolutional model compatible with real-time buffer-based processing. As our method is based on a post-training reconfiguration of the model, we show that it is able to transform models trained without causal constraints into a streaming model. We show how our method can be adapted to fit complex architectures with parallel branches. To evaluate our method, we apply it on the recent RAVE model, which provides high-quality real-time audio synthesis. We test our approach on multiple music and speech datasets and show that it is faster than overlap-add methods, while having no impact on the generation quality. Finally, we introduce two open-source implementation of our work as Max/MSP and PureData externals, and as a VST audio plugin. This allows to endow traditional digital audio workstation with real-time neural audio synthesis on a laptop CPU.

Motivation & Objective

• Enable real-time neural audio synthesis using pre-trained convolutional models that were not originally designed for streaming.
• Overcome the phase discontinuity problem in buffer-based processing of non-causal convolutions without introducing redundant computation.
• Preserve audio quality and model accuracy while enabling low-latency inference in digital audio workstations.
• Provide practical, open-source tools (Max/MSP, PureData, VST) to integrate real-time neural audio synthesis into creative audio workflows.
• Demonstrate the method's effectiveness across diverse audio datasets, including music and speech, with minimal latency and high speed.

Proposed method

• Reconfigure trained non-causal convolutional models into causal streaming versions by introducing internal delays after inference, preserving the original computational graph.
• Use cached internal states to maintain temporal continuity across audio buffer boundaries, replacing padding with state propagation.
• Apply the method post-training, requiring no retraining or architectural changes to the original model.
• Support complex architectures with parallel branches by carefully managing delay propagation across all branches.
• Implement the method as a library and integrate it into Max/MSP and PureData via custom externals, and as a VST3 plugin using the JUCE framework.
• Map latent representations to audio signals via nearest-neighbor up-sampling and provide real-time user control over latent dimensions through graphical interfaces.

Experimental results

Research questions

• RQ1 Can non-causal convolutional models be made streamable without retraining or sacrificing audio quality?
• RQ2 How can internal delays be introduced to maintain model accuracy while enabling real-time buffer-based processing?
• RQ3 To what extent does the proposed method reduce latency and improve inference speed compared to overlap-add methods?
• RQ4 Can the method be generalized to complex architectures with parallel branches and diverse audio generation tasks?
• RQ5 How effective is the method in practical creative audio environments, such as DAWs, with real-time user interaction?

Key findings

• The proposed method achieves faster inference than overlap-add methods while maintaining identical audio quality to the original non-causal model.
• The method enables real-time audio synthesis on standard laptop CPUs using open-source Max/MSP and PureData externals.
• The VST3 plugin implementation supports real-time manipulation of latent representations with user-defined scaling, bias, and noise per dimension.
• The approach successfully transforms the RAVE model into a streamable system with no degradation in audio quality across multiple music and speech datasets.
• The method is compatible with complex architectures, including those with parallel branches, due to careful delay management.
• The open-source library and plugins allow seamless integration of neural audio synthesis into traditional digital audio workstations.