[論文レビュー] Museformer: Transformer with Fine- and Coarse-Grained Attention for Music Generation

Botao Yu, Peiling Lu+7 more|arXiv (Cornell University)|Oct 19, 2022

Music Technology and Sound Studies被引用数 20

ひとこと要約

MuseformerはTransformerベースの象徴的音楽生成に対して細粒度および粗粒度の注意機構を導入し、長いシーケンスのモデリングとより良い音楽構造を実現しつつ、複雑さを低減します。構造に関連する小節を細粒度の注意に選択し、他の小節には小節要約を用いて、効率と品質を両立します。

ABSTRACT

Symbolic music generation aims to generate music scores automatically. A recent trend is to use Transformer or its variants in music generation, which is, however, suboptimal, because the full attention cannot efficiently model the typically long music sequences (e.g., over 10,000 tokens), and the existing models have shortcomings in generating musical repetition structures. In this paper, we propose Museformer, a Transformer with a novel fine- and coarse-grained attention for music generation. Specifically, with the fine-grained attention, a token of a specific bar directly attends to all the tokens of the bars that are most relevant to music structures (e.g., the previous 1st, 2nd, 4th and 8th bars, selected via similarity statistics); with the coarse-grained attention, a token only attends to the summarization of the other bars rather than each token of them so as to reduce the computational cost. The advantages are two-fold. First, it can capture both music structure-related correlations via the fine-grained attention, and other contextual information via the coarse-grained attention. Second, it is efficient and can model over 3X longer music sequences compared to its full-attention counterpart. Both objective and subjective experimental results demonstrate its ability to generate long music sequences with high quality and better structures.

研究の動機と目的

• 全自己注意の限界を超えた象徴的音楽生成における長いシーケンスのモデリングに対応する。
• 反復や長距離依存などの音楽構造をより効果的にモデル化する。
• 生成に必要な主要情報を保持しつつ、計算量とメモリ複雑性を低減する。

提案手法

• FC-Attentionを提案する：構造に関連する小節には細粒度の注意を、他の小節には要約の粗粒度の注意を適用。
• 各小節の後に要約トークンを挿入して局所的な集約を促進。
• 人間が作成した音楽の小節間の類似度統計を用いて構造に関連する小節を選択。
• FC-Attention内で2段階の要約と集約プロセスを通じてトークン表現を更新。
• 小節を小節およびビートの埋め込みで表現し、FC-Attentionを備えたTransformer風アーキテクチャを適用。
• Lakh MIDIデータセット上で perplexity と similarity error を用いて評価し、主観的聴取テストも実施。

実験結果

リサーチクエスチョン

• RQ1 Can a dual attention scheme (fine- and coarse-grained) better model long music sequences than full attention or other long-sequence transformers?
• RQ2 Do structure-related bars selected via similarity statistics improve generation of musical structure and perplexity?
• RQ3 How does Museformer scale to full-song lengths in terms of memory, speed, and quality?

主な発見

モデル	PPL (1024)	PPL (5120)	PPL (10240)	SE (%)
Music Transformer	1.66	1.77	2.55	2.49
Transformer-XL	1.64	1.45	1.43	15.66
Longformer	1.65	1.46	1.45	5.25
Linear Transformer	1.86	1.67	1.64	1.97
Museformer (ours)	1.64	1.41	1.35	0.95
w/o coarse-grained	1.65	1.42	1.38	1.08
w/o bar selection	1.65	1.43	1.39	6.39

• Museformerは比較対象モデルの中で1024、5120、10240トークン列の三つで最良の perplexity を達成。
• It yields the lowest similarity error, indicating generated music structures closely resemble human-made music.
• Subjective evaluations show Museformer scores highest on musicality, short-term structure, long-term structure, and overall preference.
• Ablation shows coarse-grained attention and bar-selection contribute to performance, with structure-related bar selection becoming more important for longer sequences.
• Museformer enables building full-song-length music with efficiency improvements over full-attention baselines (nearly linear memory growth and >3x longer sequences).