[Paper Review] MIDI-DDSP: Detailed Control of Musical Performance via Hierarchical Modeling

Yusong Wu, Ethan Manilow+7 more|arXiv (Cornell University)|Dec 17, 2021

Music Technology and Sound Studies10 citations

TL;DR

MIDI-DDSP is a hierarchical neural audio synthesis model that combines interpretable Differentiable Digital Signal Processing (DDSP) with learned priors to enable high-fidelity audio generation with detailed user control over musical performance. It models music in a 3-level hierarchy—notes, expressive performance, and synthesis parameters—allowing intervention at any level, and achieves state-of-the-art realism and control in listening tests, outperforming baselines like FluidSynth, DDSP Inference, and Ableton in perceptual preference.

ABSTRACT

Musical expression requires control of both what notes are played, and how they are performed. Conventional audio synthesizers provide detailed expressive controls, but at the cost of realism. Black-box neural audio synthesis and concatenative samplers can produce realistic audio, but have few mechanisms for control. In this work, we introduce MIDI-DDSP a hierarchical model of musical instruments that enables both realistic neural audio synthesis and detailed user control. Starting from interpretable Differentiable Digital Signal Processing (DDSP) synthesis parameters, we infer musical notes and high-level properties of their expressive performance (such as timbre, vibrato, dynamics, and articulation). This creates a 3-level hierarchy (notes, performance, synthesis) that affords individuals the option to intervene at each level, or utilize trained priors (performance given notes, synthesis given performance) for creative assistance. Through quantitative experiments and listening tests, we demonstrate that this hierarchy can reconstruct high-fidelity audio, accurately predict performance attributes for a note sequence, independently manipulate the attributes of a given performance, and as a complete system, generate realistic audio from a novel note sequence. By utilizing an interpretable hierarchy, with multiple levels of granularity, MIDI-DDSP opens the door to assistive tools to empower individuals across a diverse range of musical experience.

Motivation & Objective

• To bridge the gap between realistic neural audio synthesis and fine-grained musical control, which existing systems fail to balance.
• To create a structured, hierarchical generative model that preserves interpretability while enabling high-fidelity audio generation.
• To allow users to intervene at multiple levels—note sequence, expressive performance, or synthesis parameters—while supporting automated generation via learned priors.
• To enable assistive music creation tools that support users of all musical experience levels through interpretable, controllable audio synthesis.
• To demonstrate that learned priors over performance and synthesis can generate realistic audio from novel note sequences with accurate attribute control.

Proposed method

• MIDI-DDSP uses a 3-level hierarchy: notes (MIDI input), expressive performance (vibrato, dynamics, articulation), and synthesis parameters (DDSP-based).
• It employs three trainable modules: DDSP Inference (synthesis parameters from audio), Expression Generator (performance features from notes), and Synthesis Generator (audio from performance and synthesis parameters).
• The model leverages fixed, differentiable functions for note detection, feature extraction, and DDSP synthesis, ensuring end-to-end differentiability and interpretability.
• It uses a hierarchical autoencoder-like structure where each level is conditioned on the previous, enabling conditional generation at any stage.
• The system is trained on monophonic instrument recordings using a combination of reconstruction loss and perceptual losses to preserve audio quality and expressive detail.
• It supports both conditional generation (e.g., generate audio from notes) and disentangled control (e.g., modify vibrato independently) via inference and optimization in the latent space.

Figure 1: (Left) The MIDI-DDSP architecture. MIDI-DDSP extracts interpretable features at the performance and synthesis levels, building a modeling hierarchy by learning feature generation at each level. Red and blue components indicate encoding and decoding respectively. Shaded boxes represent modu

Experimental results

Research questions

• RQ1 Can a hierarchical model achieve both high-fidelity audio synthesis and detailed user control over musical performance?
• RQ2 How well can the model reconstruct audio from a given note sequence with expressive attributes?
• RQ3 To what extent can expressive attributes like vibrato, dynamics, and timbre be independently manipulated while preserving realism?
• RQ4 How does the model’s perceptual quality compare to existing baselines such as FluidSynth, DDSP Inference, and Ableton in human listening tests?
• RQ5 Can the model generalize to generate realistic audio from novel, unseen note sequences using learned priors?

Key findings

• In listening tests, MIDI-DDSP received a strong preference over all baselines, with 44 out of 64 participants preferring it over Ground-truth audio (p = 0.00018), indicating high perceptual realism.
• The model achieved high Pearson correlation (r > 0.7) between input control parameters and output audio attributes—e.g., 0.99 for volume and 0.96 for brightness—demonstrating strong control fidelity.
• MIDI-DDSP significantly outperformed DDSP Inference and FluidSynth in perceptual preference, with p-values < 0.01 for all pairwise comparisons except two (DDSP Inference vs. Ground-truth and MIDI2Params vs. FluidSynth).
• The system accurately predicted expressive performance attributes (e.g., vibrato, dynamics) from note sequences, with high correlation across all tested instruments, including violin, cello, and oboe.
• Independent manipulation of expressive attributes (e.g., changing vibrato or attack noise) produced perceptually plausible and realistic audio, confirming disentanglement and controllability.
• The model successfully generated high-fidelity audio from novel note sequences using trained priors, demonstrating its utility as a creative assistant in music production workflows.

Figure 2: An example of detailed user control. Given an initial generation from the full MIDI-DDSP model (top), an expert musician can adjust notes (blue), performance attributes (green), and low-level synthesis parameters (yellow) to craft a personalized expression of a musical piece (bottom).