[Paper Review] Exploring Quality and Generalizability in Parameterized Neural Audio Effects

William Mitchell, Scott H. Hawley|arXiv (Cornell University)|Jun 9, 2020

Music and Audio Processing4 citations

TL;DR

This paper improves the SignalTrain neural network for parameterized audio effect modeling by enhancing speed, accuracy, and generalizability in the waveform domain. It demonstrates that dataset restriction to single-instrument content significantly boosts model accuracy without increasing training time, while successfully modeling diverse effects including compressors and Leslie cabinet simulations at 44.1kHz/16-bit quality.

ABSTRACT

Deep neural networks have shown promise for music audio signal processing applications, often surpassing prior approaches, particularly as end-to-end models in the waveform domain. Yet results to date have tended to be constrained by low sample rates, noise, narrow domains of signal types, and/or lack of parameterized controls (i.e. "knobs"), making their suitability for professional audio engineering workflows still lacking. This work expands on prior research published on modeling nonlinear time-dependent signal processing effects associated with music production by means of a deep neural network, one which includes the ability to emulate the parameterized settings you would see on an analog piece of equipment, with the goal of eventually producing commercially viable, high quality audio, i.e. 44.1 kHz sampling rate at 16-bit resolution. The results in this paper highlight progress in modeling these effects through architecture and optimization changes, towards increasing computational efficiency, lowering signal-to-noise ratio, and extending to a larger variety of nonlinear audio effects. Toward these ends, the strategies employed involved a three-pronged approach: model speed, model accuracy, and model generalizability. Most of the presented methods provide marginal or no increase in output accuracy over the original model, with the exception of dataset manipulation. We found that limiting the audio content of the dataset, for example using datasets of just a single instrument, provided a significant improvement in model accuracy over models trained on more general datasets.

Motivation & Objective

• Address limitations in prior neural audio effect models, such as low sample rates, noise, and lack of parameterized controls.
• Enhance the SignalTrain model for professional audio workflows by achieving 44.1kHz/16-bit audio quality.
• Improve model efficiency, accuracy, and generalizability through architectural and optimization strategies.
• Demonstrate the model’s capability to generalize across diverse nonlinear audio effects, including analog and digital types.
• Investigate the impact of dataset composition—particularly single-instrument vs. mixed-instrument data—on model performance.

Proposed method

• Train the SignalTrain model in the waveform domain to preserve phase and frequency information, avoiding spectrogram-based approaches.
• Introduce trainable virtual 'knobs' to emulate parameterized controls found on analog audio equipment.
• Apply a three-pronged strategy: model speed (e.g., freezing Fourier layers), model accuracy (longer training, dataset curation), and model generalizability.
• Use larger input buffer sizes (up to 98,304 samples) to capture oscillatory characteristics of effects like Leslie cabinet chorus and tremolo.
• Implement skip connections to improve training stability and loss reduction without increasing training time.
• Evaluate model performance using validation loss and qualitative listening, with datasets from AudioMDPI and custom recordings.

Experimental results

Research questions

• RQ1 Can waveform-domain neural networks with parameterized knobs achieve high-fidelity audio effects at 44.1kHz/16-bit resolution?
• RQ2 How does restricting training data to single-instrument content affect model accuracy and generalizability?
• RQ3 What impact do architectural choices—such as frozen Fourier layers and skip connections—have on training speed and loss convergence?
• RQ4 Can the SignalTrain model generalize across diverse nonlinear audio effects, including analog Leslie cabinets and digital compressors?
• RQ5 How do input buffer size and oscillatory signal characteristics influence model training for effects like chorus and tremolo?

Key findings

• Limiting training datasets to single-instrument content significantly improved model accuracy without increasing training time, outperforming models trained on mixed-instrument data.
• The model achieved a final validation loss of 1.225e-4 on the Leslie Woofer Tremolo effect after 51.35 hours of training, indicating effective learning despite high computational cost.
• Training beyond 1,000 epochs yielded only marginal improvements in loss and no noticeable gain in audio quality, suggesting diminishing returns.
• Freezing Fourier transform layers improved training speed but caused a substantial drop in model accuracy, making the trade-off unjustifiable.
• Skip connections did not increase training time but significantly reduced loss, contributing to better convergence and model stability.
• The model successfully generalized to both analog (Leslie Cabinet) and digital (HackAudio compressors) effects, demonstrating broad applicability across diverse audio processing tasks.