[Paper Review] AccoMontage: Accompaniment Arrangement via Phrase Selection and Style Transfer

Jingwei Zhao, Gus Xia|arXiv (Cornell University)|Aug 25, 2021

Topic Modeling29 references4 citations

TL;DR

AccoMontage proposes a hybrid approach to piano accompaniment arrangement by combining phrase retrieval via dynamic programming with neural style transfer to align selected references to a lead sheet. It achieves state-of-the-art performance in coherence, structure, and musicality for full-length folk/pop songs, significantly outperforming both learning-based and template-based baselines.

ABSTRACT

Accompaniment arrangement is a difficult music generation task involving intertwined constraints of melody, harmony, texture, and music structure. Existing models are not yet able to capture all these constraints effectively, especially for long-term music generation. To address this problem, we propose AccoMontage, an accompaniment arrangement system for whole pieces of music through unifying phrase selection and neural style transfer. We focus on generating piano accompaniments for folk/pop songs based on a lead sheet (i.e., melody with chord progression). Specifically, AccoMontage first retrieves phrase montages from a database while recombining them structurally using dynamic programming. Second, chords of the retrieved phrases are manipulated to match the lead sheet via style transfer. Lastly, the system offers controls over the generation process. In contrast to pure learning-based approaches, AccoMontage introduces a novel hybrid pathway, in which rule-based optimization and deep learning are both leveraged to complement each other for high-quality generation. Experiments show that our model generates well-structured accompaniment with delicate texture, significantly outperforming the baselines.

Motivation & Objective

• To address the limitations of existing deep learning models in generating long-term, structurally coherent accompaniments for full songs.
• To improve melody-accompaniment harmony and texture consistency by leveraging existing musical references instead of generating from scratch.
• To enable controllable music generation through user-defined texture features like rhythm density and voice number.
• To unify rule-based phrase retrieval with deep learning-based style transfer for enhanced generation quality.
• To model accompaniment arrangement as a hybrid optimization and generation pipeline inspired by human compositional practices.

Proposed method

• Phrase selection is modeled as a graph optimization problem, where nodes represent candidate phrases and edges represent transitions, with node scores based on rule-based fitness and edge scores learned via contrastive learning.
• Dynamic programming is used to recombine selected phrases into a coherent sequence that respects structural constraints of the lead sheet.
• Chord manipulation via neural style transfer aligns the harmonic content of retrieved phrases to match the query lead sheet’s chord progression.
• The system employs chord-texture disentanglement to enable independent control over harmonic and textural features during re-harmonization.
• A contrastive loss is applied to ensure smooth transitions between adjacent phrases by minimizing differences in rhythm density and voice number.
• Users can pre-filter candidate phrases using two controllable texture features: rhythm density and voice number, enabling user-directed generation.

Experimental results

Research questions

• RQ1 Can a hybrid approach combining template-based retrieval and neural style transfer generate more coherent and structurally consistent accompaniments than pure learning-based methods?
• RQ2 How effectively can phrase retrieval with dynamic programming preserve long-term musical structure in full songs?
• RQ3 To what extent does style transfer improve harmony and texture fitness between reference phrases and a given lead sheet?
• RQ4 Can user-controlled features like rhythm density and voice number enhance the controllability and quality of generated accompaniments?
• RQ5 Does the proposed method outperform existing baselines in both objective metrics and subjective human evaluation?

Key findings

• AccoMontage significantly outperformed both baseline models in coherence (p < 0.05) and structure (p < 0.05) in subjective evaluations, with 72 subjects rating the generated accompaniments on a 5-point scale.
• The model achieved a mean rating of 4.3 for coherence and 4.2 for structure, indicating strong harmony and structural alignment with the melody.
• The contrastive loss showed a consistent decreasing trend from Random to Same Song to Adjacent phrase pairs, demonstrating reliable detection of smooth transitions.
• The ranking accuracy (Rank@50) was 0.2425 for phrase accuracy and 0.3769 for song accuracy, indicating strong performance in selecting correct adjacent phrases from a pool of candidates.
• The system achieved a marginally better performance in musicality (p = 0.053), suggesting high overall musical quality despite not reaching strict statistical significance.
• The results confirm that combining rule-based phrase retrieval with deep learning-based style transfer enables high-quality, controllable, and long-term accompaniment generation for full songs.