[Paper Review] MeloForm: Generating Melody with Musical Form based on Expert Systems and Neural Networks

Peiling Lu, Xu Tan+4 more|arXiv (Cornell University)|Aug 30, 2022

Music Technology and Sound Studies4 citations

TL;DR

MeloForm is a hybrid system that generates melodies with precise musical form control by combining expert systems for structural composition with a Transformer-based neural network for melodic refinement. It achieves 97.79% accuracy in musical form control and outperforms baselines in subjective evaluation by 0.75–0.89 in structure, thematic, richness, and overall quality, without requiring labeled musical form data.

ABSTRACT

Human usually composes music by organizing elements according to the musical form to express music ideas. However, for neural network-based music generation, it is difficult to do so due to the lack of labelled data on musical form. In this paper, we develop MeloForm, a system that generates melody with musical form using expert systems and neural networks. Specifically, 1) we design an expert system to generate a melody by developing musical elements from motifs to phrases then to sections with repetitions and variations according to pre-given musical form; 2) considering the generated melody is lack of musical richness, we design a Transformer based refinement model to improve the melody without changing its musical form. MeloForm enjoys the advantages of precise musical form control by expert systems and musical richness learning via neural models. Both subjective and objective experimental evaluations demonstrate that MeloForm generates melodies with precise musical form control with 97.79% accuracy, and outperforms baseline systems in terms of subjective evaluation score by 0.75, 0.50, 0.86 and 0.89 in structure, thematic, richness and overall quality, without any labelled musical form data. Besides, MeloForm can support various kinds of forms, such as verse and chorus form, rondo form, variational form, sonata form, etc.

Motivation & Objective

• To address the challenge of generating melodies with explicit, high-level musical forms (e.g., verse-chorus, sonata) in data-driven music generation.
• To overcome the limitations of purely neural models, which implicitly learn repetition but lack precise form control.
• To integrate expert systems for accurate structural composition with neural networks for enhanced melodic expressiveness.
• To eliminate the need for costly human-annotated musical form data by generating synthetic, perfectly labeled form data via expert rules.
• To enable flexible, form-agnostic melody generation across diverse musical forms such as rondo, variational, and sonata forms.

Proposed method

• An expert system generates melodies by hierarchically developing motifs into phrases and sections, applying repetitions and variations according to a pre-specified musical form.
• A Transformer-based refinement model improves melodic richness by conditioning on fine-grained rhythm and harmony patterns while preserving the original musical form.
• The refinement strategy operates at the phrase level, using iterative refinement with section differentiation tokens (e.g., 'AVGPITCH', 'SPAN') to modulate tension and expression.
• Musical form data is generated synthetically by the expert system, providing zero-cost, perfectly accurate labels for training and evaluation.
• The system supports various musical forms by adapting structural construction and variation rules—e.g., transposition for recapitulation in sonata form or motif development for variational forms.
• The neural refinement model is conditioned on tonality, rhythm, and section-level features to ensure form-consistent, musically coherent output.

Experimental results

Research questions

• RQ1 Can a hybrid expert system and neural network approach achieve precise, high-level musical form control in melody generation without labeled data?
• RQ2 How effective is the combination of rule-based structural composition and neural refinement in improving melodic richness and expressiveness?
• RQ3 To what extent does the proposed method outperform purely data-driven models in subjective and objective evaluations of musical quality?
• RQ4 Can the system generalize across diverse musical forms such as verse-chorus, rondo, variational, and sonata forms?
• RQ5 What components of the refinement pipeline (e.g., rhythm conditioning, iterative refinement, section differentiation) contribute most to perceptual quality?

Key findings

• MeloForm achieves 97.79% accuracy in musical form control, demonstrating precise and reliable structural generation without any labeled data.
• In subjective evaluation, MeloForm outperforms baseline systems by 0.75, 0.50, 0.86, and 0.89 points in structure, thematic, richness, and overall quality, respectively.
• The ablation study confirms that the expert system and neural refinement components are both essential, with significant performance drops when either is removed.
• The inclusion of fine-grained rhythm conditioning and section differentiation tokens significantly improves perceptual quality, as shown in preference-based evaluations.
• The system successfully generalizes across multiple musical forms, including verse-chorus, rondo, variational, and sonata forms, with consistent structural and expressive quality.
• The authors release synthetic melody datasets and code, enabling future research on music form modeling and structured generation.