[Paper Review] Multi-instrument Music Synthesis with Spectrogram Diffusion

Curtis Hawthorne, Ian Simon+5 more|arXiv (Cornell University)|Jun 11, 2022

Music and Audio Processing15 citations

TL;DR

This paper proposes a real-time, multi-instrument music synthesis system using a spectrogram diffusion model that converts MIDI sequences into high-fidelity audio with full note-level control. By combining a Transformer-based MIDI-to-spectrogram decoder trained with Denoising Diffusion Probabilistic Models (DDPM) and a GAN-based spectrogram inverter, the method achieves superior audio quality and temporal consistency over segment boundaries compared to autoregressive baselines.

ABSTRACT

An ideal music synthesizer should be both interactive and expressive, generating high-fidelity audio in realtime for arbitrary combinations of instruments and notes. Recent neural synthesizers have exhibited a tradeoff between domain-specific models that offer detailed control of only specific instruments, or raw waveform models that can train on any music but with minimal control and slow generation. In this work, we focus on a middle ground of neural synthesizers that can generate audio from MIDI sequences with arbitrary combinations of instruments in realtime. This enables training on a wide range of transcription datasets with a single model, which in turn offers note-level control of composition and instrumentation across a wide range of instruments. We use a simple two-stage process: MIDI to spectrograms with an encoder-decoder Transformer, then spectrograms to audio with a generative adversarial network (GAN) spectrogram inverter. We compare training the decoder as an autoregressive model and as a Denoising Diffusion Probabilistic Model (DDPM) and find that the DDPM approach is superior both qualitatively and as measured by audio reconstruction and Fréchet distance metrics. Given the interactivity and generality of this approach, we find this to be a promising first step towards interactive and expressive neural synthesis for arbitrary combinations of instruments and notes.

Motivation & Objective

• To develop a real-time, interactive neural synthesizer capable of generating high-fidelity audio for arbitrary combinations of instruments and notes.
• To overcome the trade-off between control and generality in existing neural synthesizers by using a generic, data-efficient architecture.
• To improve audio quality and temporal coherence in spectrogram-based synthesis by replacing autoregressive decoding with diffusion modeling.
• To enable consistent audio generation across segment boundaries through context-aware conditioning on previously generated spectrograms.
• To demonstrate the feasibility of training a single model on diverse paired audio/MIDI datasets, including real recordings and synthetic data.

Proposed method

• The model uses a two-stage pipeline: first, a Transformer encoder-decoder converts MIDI note events into spectrograms, with the decoder trained as a Denoising Diffusion Probabilistic Model (DDPM).
• The DDPM decoder iteratively denoises Gaussian noise to generate high-fidelity spectrograms, improving quality over frame-wise autoregressive decoding.
• To ensure temporal continuity across 5-second audio segments, a second encoder stack conditions the decoder on the spectrogram of the previously generated segment.
• At inference, the generated spectrogram is inverted to a waveform using a GAN-based vocoder similar to MelGAN.
• The model is trained on a wide range of paired audio/MIDI datasets, including real recordings (e.g., Guitarset, URMP) and synthetic data, enabling generalization across instruments and acoustic settings.
• Model capacity is scaled up to a 'base' architecture to handle the additional context input, reducing boundary artifacts and improving audio quality.

Experimental results

Research questions

• RQ1 Can a single, generic Transformer-based model effectively synthesize high-fidelity audio for arbitrary instrument combinations from MIDI input?
• RQ2 Does training the spectrogram decoder with DDPM outperform autoregressive decoding in terms of audio quality and perceptual consistency?
• RQ3 Can context conditioning on prior spectrogram segments effectively eliminate audible artifacts at segment boundaries in continuous audio generation?
• RQ4 To what extent can a unified model trained on diverse datasets generalize to real-world recordings, including subtle articulations like fret noise and vibrato?
• RQ5 How does the model’s performance compare quantitatively to autoregressive baselines using standard metrics like Fréchet Audio Distance and Transcription F1?

Key findings

• The DDPM-based decoder outperformed autoregressive baselines in both qualitative and quantitative evaluations, achieving the highest Fréchet Audio Distance (FAD) and Transcription F1 scores.
• The addition of context conditioning on prior spectrograms eliminated segment boundary artifacts, significantly improving temporal coherence in long-form audio generation.
• The model achieved a Transcription F1 score of nearly 1.0 on the validation set, indicating near-perfect reconstruction of the target spectrograms, though audio fidelity remained below raw waveform quality.
• The model synthesized natural-sounding articulations such as fret noise in guitar recordings and vibrato in orchestral instruments, even without explicit velocity or articulation conditioning.
• Despite using a spectrogram inversion step, the model remained slightly faster than real-time inference, enabling interactive applications.
• No overfitting was observed on validation sets, suggesting that larger models could further improve audio fidelity.