[Paper Review] Music Transformer

Cheng-Zhi Anna Huang, Ashish Vaswani+8 more|arXiv (Cornell University)|Sep 12, 2018

Music and Audio Processing48 citations

TL;DR

A Transformer model with memory-efficient relative attention that captures long-term musical structure, enabling long sequences and improved quality, evaluated on JSB Chorales and Piano-e-Competition.

ABSTRACT

Music relies heavily on repetition to build structure and meaning. Self-reference occurs on multiple timescales, from motifs to phrases to reusing of entire sections of music, such as in pieces with ABA structure. The Transformer (Vaswani et al., 2017), a sequence model based on self-attention, has achieved compelling results in many generation tasks that require maintaining long-range coherence. This suggests that self-attention might also be well-suited to modeling music. In musical composition and performance, however, relative timing is critically important. Existing approaches for representing relative positional information in the Transformer modulate attention based on pairwise distance (Shaw et al., 2018). This is impractical for long sequences such as musical compositions since their memory complexity for intermediate relative information is quadratic in the sequence length. We propose an algorithm that reduces their intermediate memory requirement to linear in the sequence length. This enables us to demonstrate that a Transformer with our modified relative attention mechanism can generate minute-long compositions (thousands of steps, four times the length modeled in Oore et al., 2018) with compelling structure, generate continuations that coherently elaborate on a given motif, and in a seq2seq setup generate accompaniments conditioned on melodies. We evaluate the Transformer with our relative attention mechanism on two datasets, JSB Chorales and Piano-e-Competition, and obtain state-of-the-art results on the latter.

Motivation & Objective

• Demonstrate that Transformers can generate music with long-range, repeating structure across multiple timescales.
• Extend the Transformer with relative timing (and optional pitch) information to improve modeling of musical relations.
• Reduce the memory cost of relative attention to enable training on long sequences.
• Showcase sequencing representations for music (score-like and performance-like data) and assess both unconditioned generation and melody-conditioned accompaniment.

Proposed method

• Introduce a memory-efficient relative self-attention mechanism that reduces intermediate memory from O(L^2 D) to O(LD) and a skewing procedure to align relative logits.
• Represent music as token sequences using dataset-appropriate encodings (JSB Chorales as a 16th-note grid; Piano-e-Competition as MIDI-like event-based tokens).
• Extend relative attention to include timing (and optionally pitch) relations between positions, yielding S^rel logits integrated into attention.
• Experiment with global and local relative attention on JSB Chorales and Piano-e-Competition, comparing to baselines and prior models.
• Conduct human listening tests and analyze priming/conditioning by melody to assess coherence and musicality.

Experimental results

Research questions

• RQ1 Does relative self-attention improve perplexity and sample quality over the baseline Transformer on symbolic music datasets?
• RQ2 Can the model generate coherent long-term musical structure and continuations longer than training sequences?
• RQ3 Does incorporating timing (and pitch) relations yield better performance and generalization?
• RQ4 How does the model perform in a seq2seq setting to generate accompaniments conditioned on melodies?

Key findings

• Relative attention improves negative log-likelihood and sample coherence over the baseline Transformer on the JSB Chorales dataset.
• On Piano-e-Competition, the Transformer with relative attention achieves state-of-the-art perplexity and outperforms baseline models.
• The memory-efficient implementation enables training on long sequences (thousands of steps) by reducing intermediate memory usage, enabling longer compositions.
• Relatively attended models demonstrate better priming and continuation of motifs, preserving long-term structure and regular phrasing.
• In a melody-conditioned setup, the relative Transformer yields better conditional NLL than the baseline.
• Human evaluations show perceptual improvements in musicality for the relative-attention model compared with the baseline.