[Paper Review] An Improved Relative Self-Attention Mechanism for Transformer with Application to Music Generation

Cheng-Zhi Anna Huang, Ashish Vaswani+6 more|arXiv (Cornell University)|Sep 12, 2018

Music and Audio Processing11 references57 citations

TL;DR

This paper proposes an improved relative self-attention mechanism that reduces memory complexity from $O(l^2d)$ to $O(ld)$, enabling efficient training on longer musical sequences. The method enhances long-term structural modeling in music generation, yielding higher-quality samples and coherent continuations even beyond training sequence lengths.

ABSTRACT

Music relies heavily on self-reference to build structure and meaning. We explore the Transformer architecture (Vaswani et al., 2017) as a generative model for music, as self-attention has shown compelling results on tasks that require long-term structure such as Wikipedia summary generation (Liu et al, 2018). However, timing information is critical for polyphonic music, and Transformer does not explicitly model absolute or relative timing in its structure. To address this challenge, Shaw et al. (2018) introduced relative position representations to self-attention to improve machine translation. However, the formulation was not scalable to longer sequences. We propose an improved formulation which reduces the memory requirements of the relative position computation from $O(l^2d)$ to $O(ld)$, making it possible to train much longer sequences and achieve faster convergence. In experiments on symbolic music we find that relative self-attention substantially improves sample quality for unconditioned generation and is able to generate sequences of lengths longer than those from the training set. When primed with an initial sequence, the model generates continuations that develop the prime coherently and exhibit long-term structure. Relative self-attention can be instrumental in capturing richer relationships within a musical piece.

Motivation & Objective

• To address the lack of explicit timing modeling in standard Transformers for music generation.
• To improve long-term structural modeling in symbolic music generation using self-attention.
• To scale relative position representations to longer sequences by reducing memory complexity.
• To enable coherent, high-quality unconditional and conditional music generation with long-range dependencies.

Proposed method

• Introduces a reformulated relative self-attention mechanism that reduces memory requirements from $O(l^2d)$ to $O(ld)$.
• Adapts relative position representations originally proposed by Shaw et al. (2018) for improved scalability.
• Maintains the ability to model relative positional relationships between tokens in long sequences.
• Applies the improved mechanism within a Transformer-based generative model for symbolic music.
• Uses the mechanism to capture long-range dependencies critical for musical structure and coherence.

Experimental results

Research questions

• RQ1 Can relative self-attention be made scalable to longer musical sequences without prohibitive memory costs?
• RQ2 Does the improved relative self-attention mechanism enhance sample quality in unconditional music generation?
• RQ3 Can the model generate coherent musical continuations when primed with an initial sequence?
• RQ4 To what extent does the mechanism capture long-term structural relationships in music?

Key findings

• The improved relative self-attention mechanism reduces memory usage from $O(l^2d)$ to $O(ld)$, enabling training on significantly longer sequences.
• The model generates higher-quality music samples in unconditional generation compared to standard self-attention.
• The model produces coherent musical continuations when conditioned on an initial sequence, maintaining structural coherence.
• The model successfully generates sequences longer than those in the training set, demonstrating robust long-range modeling.