[Paper Review] Towards democratizing music production with AI-Design of Variational Autoencoder-based Rhythm Generator as a DAW plugin

Nao Tokui|arXiv (Cornell University)|Apr 1, 2020

Music and Audio Processing15 references4 citations

TL;DR

This paper presents M4L.RhythmVAE, a VAE-based rhythm generator plugin for Ableton Live that enables musicians to train custom rhythm models by drag-and-drop MIDI files and explore generative outputs via real-time control of a 2D latent space using an XY pad. Professional musicians found it useful for creative exploration, demonstrating a practical path toward democratizing AI in music production.

ABSTRACT

There has been significant progress in the music generation technique utilizing deep learning. However, it is still hard for musicians and artists to use these techniques in their daily music-making practice. This paper proposes a Variational Autoencoder\cite{Kingma2014}(VAE)-based rhythm generation system, in which musicians can train a deep learning model only by selecting target MIDI files, then generate various rhythms with the model. The author has implemented the system as a plugin software for a DAW (Digital Audio Workstation), namely a Max for Live device for Ableton Live. Selected professional/semi-professional musicians and music producers have used the plugin, and they proved that the plugin is a useful tool for making music creatively. The plugin, source code, and demo videos are available online.

Motivation & Objective

• To address the gap between advanced AI music generation and practical use by musicians who lack technical expertise in deep learning.
• To provide a user-friendly, interactive tool that allows non-experts to train and use custom rhythm generation models within a standard DAW workflow.
• To enable direct, real-time manipulation of the latent space for creative exploration of rhythm patterns.
• To demonstrate that musicians can generate novel, usable musical ideas using AI tools trained on their own preferred MIDI data.

Proposed method

• The system implements a Variational Autoencoder (VAE) trained on user-selected MIDI files conforming to the General MIDI standard, focusing on drum patterns from channel 10.
• MIDI note onsets are mapped to 9 predefined drum sounds using a fixed GM drum mapping, with rhythmic patterns encoded as 16th-note resolution sequences.
• The VAE uses a 2D latent space to enable intuitive control via a standard XY pad interface in Max for Live, allowing real-time generation of rhythm patterns.
• The plugin integrates with Ableton Live’s sequencer, synchronizing generated rhythms and enabling automation of latent vector movements for musical progression creation.
• Training is initiated by clicking a button after drag-and-drop of MIDI files, with loss curves visualized in real time for monitoring convergence.
• The model is deployed using TensorFlow.js within a Node.js runtime for Max, enabling on-device inference without external dependencies.

Experimental results

Research questions

• RQ1 Can musicians without deep learning expertise effectively train and use a custom rhythm generation model within a standard DAW environment?
• RQ2 How does direct, real-time control of a 2D latent space influence creative exploration and idea generation in music production?
• RQ3 To what extent can a small, user-trained VAE model with a 2D latent space generate diverse and musically meaningful rhythm patterns?
• RQ4 Can such a plugin serve as a practical tool for generating novel musical ideas in real-world music production workflows?

Key findings

• Professional musicians and producers who tested the plugin reported it as a useful tool for generating new musical ideas, with one participant explicitly stating it inspired fresh creativity.
• The plugin enabled real-time, interactive rhythm generation: as users moved the XY pad, new rhythm patterns were instantly generated and played back in sync with Ableton Live’s sequencer.
• Participants successfully used the plugin in live performance settings, with documented stage performances using AI-generated rhythms, demonstrating real-world usability.
• The model’s 2D latent space, while limiting in capacity, was sufficient for meaningful exploration and variation when trained on user-specific MIDI data, supporting the hypothesis that domain-specific models outperform generic ones for creative use.
• The ability to record and automate latent vector movements allowed users to create evolving rhythmic progressions, demonstrating integration with standard DAW workflows.
• The source code, plugin, and demo videos are publicly available, supporting reproducibility and broader adoption.