[Paper Review] Performing Structured Improvisations with pre-trained Deep Learning Models

Pablo Samuel Castro|arXiv (Cornell University)|Apr 30, 2019

Music Technology and Sound Studies4 citations

TL;DR

This paper presents a real-time, style-preserving system that integrates pre-trained deep learning models (MelodyRNN and DrumsRNN) into live musical improvisation by using human input as a primer. The system enables immediate, responsive, and stylistically coherent music generation without requiring machine learning expertise, successfully maintaining the performer’s personal style in live jazz settings.

ABSTRACT

The quality of outputs produced by deep generative models for music have seen a dramatic improvement in the last few years. However, most deep learning models perform in "offline" mode, with few restrictions on the processing time. Integrating these types of models into a live structured performance poses a challenge because of the necessity to respect the beat and harmony. Further, these deep models tend to be agnostic to the style of a performer, which often renders them impractical for live performance. In this paper we propose a system which enables the integration of out-of-the-box generative models by leveraging the musician's creativity and expertise.

Motivation & Objective

• To address the challenge of integrating high-quality, long-inference-time generative AI models into real-time, structured musical performances.
• To enable live improvisation with pre-trained models while preserving the stylistic identity of the human performer.
• To eliminate the need for machine learning expertise in using advanced generative models for music performance.
• To reduce latency and improve timing control in AI-assisted improvisation through real-time human input processing.
• To explore how AI-generated improvisations can push expert musicians beyond predictable patterns.

Proposed method

• The system uses a human performer’s real-time MIDI input as a primer sequence for pre-trained MelodyRNN and DrumsRNN models.
• It processes the performer’s notes in real time, with no perceptible lag, to generate new melodic and rhythmic content.
• The model generates new notes based on the statistical distribution over pitch and duration tokens, conditioned on the human input sequence.
• The system avoids 16th-note quantization by using continuous-time input, allowing for more natural rhythmic expression.
• A MIDI foot pedal is introduced to give the performer control over when the AI begins replacing their input.
• The system outputs MIDI, enabling integration with any sound module or synthesizer for expressive, organic sound.

Experimental results

Research questions

• RQ1 Can pre-trained deep generative models be used in real-time, live musical improvisation without significant latency?
• RQ2 How can the stylistic identity of a human performer be preserved when integrating AI-generated content?
• RQ3 Can a system be designed to require no machine learning expertise while still enabling high-quality, responsive AI-assisted improvisation?
• RQ4 How does the introduction of AI-generated content affect the creative process and musical exploration of expert performers?
• RQ5 What control mechanisms can improve the timing and musical coherence of AI-assisted improvisation?

Key findings

• The system achieved real-time performance with no noticeable latency between the performer’s input and the AI-generated output.
• Audience members, familiar with the performer’s style, did not detect the AI’s involvement, indicating strong stylistic consistency.
• Performers reported being pushed into new creative spaces due to the uncertainty of AI-generated notes, reducing reliance on habitual lines.
• The system worked best in modal jazz with stable harmony; harmonic shifts caused timing and voice-leading issues.
• The introduction of a MIDI foot pedal improved control over when the AI began replacing the performer’s notes, reducing awkward transitions.
• The system successfully maintained the performer’s musical identity while generating novel, coherent improvisations.