[Paper Review] DDSP: Differentiable Digital Signal Processing

Jesse Engel, Lamtharn Hantrakul+2 more|arXiv (Cornell University)|Jan 14, 2020

Music and Audio Processing37 references77 citations

TL;DR

The paper introduces the DDSP library that integrates differentiable DSP components (oscillators, envelopes, filters, and reverberation) with neural networks to achieve high-fidelity audio synthesis with interpretable, modular control over pitch, loudness, and timbre without heavy autoregressive or adversarial training.

ABSTRACT

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library is publicly available at https://github.com/magenta/ddsp and we welcome further contributions from the community and domain experts.

Motivation & Objective

• Motivate and enable end-to-end learning with strong inductive biases from classic DSP for audio synthesis.
• Develop a modular, differentiable toolkit (DDSP) that combines oscillators, envelopes, filters, and reverb with neural networks.
• Demonstrate that DDSP enables independent control of pitch and loudness, extrapolation to unseen pitches, and timbre transfer.
• Show that DDSP can achieve high-quality synthesis with smaller models compared to autoregressive or GAN-based baselines.

Proposed method

• Implement differentiable oscillator-based additive synthesis using time-varying fundamental frequency f0(n) and harmonic amplitudes A(n) with A_k(n)=A(n)c_k(n).
• Use envelopes and smoothing to upsample slow neural frame rates to audio rate while avoiding artifacts.
• Design time-varying linear-phase FIR filters via the frequency sampling method with network-predicted transfer functions H_l per frame.
• Combine additive (harmonic) and filtered noise (subtractive) synthesis for a Harmonic plus Noise model.
• Incorporate a differentiable reverb via frequency-domain convolution to model long impulse responses.
• Train an autoencoder with a multi-scale spectral loss (L_i = ||S_i - S_i_hat||_1 + alpha ||log S_i - log S_i_hat||_1) over multiple FFT sizes.

Experimental results

Research questions

• RQ1 Can differentiable DSP components enable end-to-end learning for high-fidelity audio synthesis without autoregressive or adversarial losses?
• RQ2 Does a modular DDSP architecture allow independent control of pitch, loudness, and timbre while supporting extrapolation to unseen conditions?
• RQ3 Can explicit modeling of room acoustics (reverb) be separated from source generation to enable tasks like blind dereverberation and acoustic transfer?
• RQ4 Is a compact DDSP-based autoencoder competitive with state-of-the-art neural vocoders in quality and efficiency?
• RQ5 How do DDSP components contribute to interpretable and controllable music/audio generation?

Key findings

• DDSP autoencoders can resynthesize solo violin and NSynth-style data with high fidelity using far smaller models than WaveRNN and similar baselines.
• Supervised DDSP outperforms WaveRNN on F0 L1 error and achieves lower loudness error than the compared model on NSynth.
• Unsupervised DDSP with a perceptual CREPE loss learns meaningful F0 and timbre without explicit pitch conditioning, outperforming some baselines.
• Independent control over pitch and loudness is demonstrated through separate conditioning of f(t) and l(t), with z(t) encoding timbre, and interpolations showing smooth perceptual transitions.
• Dereverberation and acoustic transfer are achieved by separating room impulse responses and applying learned reverb to new audio, enabling blind dereverberation and environment transfer.
• Timbre transfer from singing to violin is demonstrated by conditioning on F0/Loudness from the voice and reverb transfer from violin recordings.