[Paper Review] MMM : Exploring Conditional Multi-Track Music Generation with the Transformer

Jeffrey Ens, Philippe Pasquier|arXiv (Cornell University)|Aug 13, 2020

Music Technology and Sound Studies15 references38 citations

TL;DR

MMM introduces a Transformer-based system for conditional multi-track music generation using a novel per-track token sequence representation, enabling track-level and bar-level inpainting as well as per-track attribute control.

ABSTRACT

We propose the Multi-Track Music Machine (MMM), a generative system based on the Transformer architecture that is capable of generating multi-track music. In contrast to previous work, which represents musical material as a single time-ordered sequence, where the musical events corresponding to different tracks are interleaved, we create a time-ordered sequence of musical events for each track and concatenate several tracks into a single sequence. This takes advantage of the Transformer's attention-mechanism, which can adeptly handle long-term dependencies. We explore how various representations can offer the user a high degree of control at generation time, providing an interactive demo that accommodates track-level and bar-level inpainting, and offers control over track instrumentation and note density.

Motivation & Objective

• Improve practical usability of generative music by enabling multi-track generation with fine-grained user control.
• Develop a novel representation that preserves track-level conditioning while leveraging Transformer attention for long-range dependencies.
• Enable track-level and bar-level inpainting and explicit instrument and note-density control per track.

Proposed method

• Represent multi-track music as a sequence of tokens per track and concatenate tracks into a single sequence (MultiTrack representation).
• Introduce BarFill representation by replacing target bars with fill tokens and appending the actual bars after generation for bar-level inpainting.
• Train GPT-2 style Transformer models (MMMBar on BarFill; MMMTrack on MultiTrack) with 4-bar and 8-bar configurations, using Lahk MIDI Dataset (LMD).
• Use 128 NOTE_ON tokens, 128 NOTE_OFF tokens, and 48 TIME_SHIFT tokens per bar, with BAR_START/BAR_END and TRACK_START/TRACK_END metadata tokens, plus INSTRUMENT and DENSITY_LEVEL tokens per track.
• Enable interactive generation modes including track inpainting, bar inpainting, and per-track instrument and density attribute control, plus iterative generation and meta-algorithms for refinement.

Experimental results

Research questions

• RQ1 How can a Transformer model generate and condition multi-track symbolic music effectively without interleaving track events?
• RQ2 Can track-level and bar-level inpainting be achieved within a single model while maintaining controllable instrument and note density per track?
• RQ3 What representations and token schemas best support fine-grained control over multi-track music generation?
• RQ4 How does the MMM approach leverage Transformer attention to handle long-range dependencies across tracks?

Key findings

• A novel MultiTrack/BarFill token-based representation enables track-level and bar-level inpainting and per-track attribute control.
• MMMBar and MMMTrack models trained on 4-bar and 8-bar segments demonstrate flexible generation with conditional dependencies across tracks.
• The BarFill representation allows conditioning on specified bars by inserting fill placeholders that are later filled, enabling bar-level inpainting.
• The system supports instrument-level control for each track via INSTRUMENT tokens and note-density control via DENSITY_LEVEL tokens.
• The authors note that ~99.8% of 10-track 4-bar, 86.8% of 10-track 8-bar, and 38.8% of 10-track 16-bar segments in LMD can be represented with <2048 tokens, suggesting practical scalability.