[Paper Review] Learning to Groove with Inverse Sequence Transformations

Jon Gillick, Adam P. Roberts+3 more|arXiv (Cornell University)|May 14, 2019

Teaching and Learning Programming40 citations

TL;DR

The paper develops Seq2Seq and Variational Information Bottleneck models to translate drum scores into expressive performances, introducing the Groove MIDI Dataset and tasks like Humanization, Infilling, and Tap2Drum.

ABSTRACT

We explore models for translating abstract musical ideas (scores, rhythms) into expressive performances using Seq2Seq and recurrent Variational Information Bottleneck (VIB) models. Though Seq2Seq models usually require painstakingly aligned corpora, we show that it is possible to adapt an approach from the Generative Adversarial Network (GAN) literature (e.g. Pix2Pix (Isola et al., 2017) and Vid2Vid (Wang et al. 2018a)) to sequences, creating large volumes of paired data by performing simple transformations and training generative models to plausibly invert these transformations. Music, and drumming in particular, provides a strong test case for this approach because many common transformations (quantization, removing voices) have clear semantics, and models for learning to invert them have real-world applications. Focusing on the case of drum set players, we create and release a new dataset for this purpose, containing over 13 hours of recordings by professional drummers aligned with fine-grained timing and dynamics information. We also explore some of the creative potential of these models, including demonstrating improvements on state-of-the-art methods for Humanization (instantiating a performance from a musical score).

Motivation & Objective

• Create a large, aligned drum performance dataset with precise timing and dynamics (Groove MIDI Dataset) to enable expressive performance modeling.
• Develop and evaluate models that translate simplified drum representations into realistic performances (Humanization).
• Introduce and study additional tasks—Drum Infilling and Tap2Drum—to enable user-friendly control of drum performances.
• Propose the GrooVAE family of models for expressive sequence generation and analyze their perceptual quality.
• Advance methods for learning inverse sequence transformations in sequential data like music.

Proposed method

• Adapt Seq2Seq architectures to map from compressed musical representations to detailed drum performances (hits, offsets, velocities).
• Represent timing with continuous Gaussian offsets and velocities, predicting H, V, O at 16th-note resolutions.
• Train with teacher forcing and use a multi-component loss combining hit prediction, velocity error, and offset error (Equation L_t).
• Incorporate a Groove Transfer variant to disentangle groove (performance) from score content for style transfer.
• Apply a Variational Information Bottleneck (VIB) to embeddings to balance realism and control (ELBO with beta=0.2).
• Provide baselines (Quantized, Linear Regression, KNN) and several neural models (MLP, Seq2Seq, Groove Transfer) for comparison.

Experimental results

Research questions

• RQ1 Can an inverse sequence transformation approach generate realistic drum performances from simplified representations?
• RQ2 Do Seq2Seq and VIB-based models outperform traditional baselines in Humanization of drum scores?
• RQ3 How well can models perform Infilling and Tap2Drum tasks, and are outputs perceptually competitive with real data?
• RQ4 Does Groove Transfer enable effective drum performance style transfer without sacrificing realism?

Key findings

• Seq2Seq with VIB achieves the best perceptual and quantitative performance for Humanization among tested models.
• Listeners preferred Seq2Seq (with VIB) over KNN baselines in head-to-head tests and found outputs competitive with real data.
• Quantitative metrics show Seq2Seq (+VIB) achieves MAE 18.47 ms and MSE 0.028 (16th note), outperforming many baselines.
• Groove Transfer provides meaningful groove/style control but generally trails Seq2Seq on timing accuracy.
• Infilling can produce outputs that listeners rate as more human than real data in some cases, suggesting a potential corrective tool use.
• Tap2Drum outputs are slightly less preferred than real data but still viable for control-based improvisation.