[Paper Review] Universal audio synthesizer control with normalizing flows

Philippe Esling, Naotake Masuda+3 more|arXiv (Cornell University)|Jul 1, 2019

Music Technology and Sound StudiesComputer Science16 references34 citations

TL;DR

The paper formulates synthesizer control as learning an organized latent audio space mapped invertibly to the parameter space, using VAEs with normalizing flows and introducing regression and disentangling flows to enable parameter inference, macro-controls, and audio-based preset exploration.

ABSTRACT

The ubiquity of sound synthesizers has reshaped music production and even entirely defined new music genres. However, the increasing complexity and number of parameters in modern synthesizers make them harder to master. Hence, the development of methods allowing to easily create and explore with synthesizers is a crucial need. Here, we introduce a novel formulation of audio synthesizer control. We formalize it as finding an organized latent audio space that represents the capabilities of a synthesizer, while constructing an invertible mapping to the space of its parameters. By using this formulation, we show that we can address simultaneously automatic parameter inference, macro-control learning and audio-based preset exploration within a single model. To solve this new formulation, we rely on Variational Auto-Encoders (VAE) and Normalizing Flows (NF) to organize and map the respective auditory and parameter spaces. We introduce the disentangling flows, which allow to perform the invertible mapping between separate latent spaces, while steering the organization of some latent dimensions to match target variation factors by splitting the objective as partial density evaluation. We evaluate our proposal against a large set of baseline models and show its superiority in both parameter inference and audio reconstruction. We also show that the model disentangles the major factors of audio variations as latent dimensions, that can be directly used as macro-parameters. We also show that our model is able to learn semantic controls of a synthesizer by smoothly mapping to its parameters. Finally, we discuss the use of our model in creative applications and its real-time implementation in Ableton Live

Motivation & Objective

• Motivate an organized latent representation of a synthesizer's audio capabilities.
• Provide an invertible mapping between latent audio space and synthesis parameter space.
• Enable simultaneous parameter inference, macro-control learning, and audio-based preset exploration.
• Introduce regression flows and disentangling flows to map and organize latent factors.
• Demonstrate improved audio reconstruction and parameter inference over baselines.

Proposed method

• Formalize synthesizer control as learning two latent spaces connected by an invertible mapping.
• Use a VAE to learn an organized latent audio space z, combined with Normalizing Flows to increase posterior expressiveness.
• Define a regression flow to map latent z to synthesis parameters v with an additive Gaussian noise model.
• Introduce Flow_post and Flow_cond variants to optimize the mapping and its uncertainty.
• Extend the model with disentangling flows to align latent dimensions with semantic tags t (supervised where available).
• Train on a Diva synthesizer dataset of paired audio and MIDI-controllable parameter sets; evaluate against baselines on parameter inference and audio reconstruction.

Experimental results

Research questions

• RQ1 Can an organized latent audio space mapped invertibly to the parameter space improve parameter inference and audio reconstruction?
• RQ2 Do regression and disentangling flows enable effective macro-control learning and semantic dimensions for perceptual control?
• RQ3 Is the proposed approach robust to higher parameter counts and out-of-domain audio?
• RQ4 Can audio-based neighborhood exploration be used to navigate presets via the latent space?
• RQ5 How does the method perform in real-time application contexts (e.g., Ableton Live)?

Key findings

• The Flow_reg model achieves the best audio reconstruction performance among evaluated methods.
• AE-based models (including Flow variants) better captured audio structure than direct parameter regression baselines, even when parameter inference was less accurate.
• Increasing the number of parameters from 16 to 32 degrades baseline methods more than flows, with Flow variants showing strongest resilience to higher dimensional parameter spaces.
• Disentangling flows provide explicit semantic dimensions useful for macro-controls, though they may slightly reduce raw audio fidelity compared to Flow_reg.
• Latent audio space encodings yielded meaningful neighborhoods; parameter decoding from this space preserved audio structure better than direct parameter inference in some cases.