[Paper Review] Automatic Melody Harmonization with Triad Chords: A Comparative Study

Yin-Cheng Yeh, Wen-Yi Hsiao+8 more|arXiv (Cornell University)|Jan 8, 2020

Music and Audio Processing44 references7 citations

TL;DR

This paper presents a comparative evaluation of canonical melody harmonization methods—ranging from template matching and HMMs to genetic algorithms and deep learning—on a newly collected dataset of 9,226 melody/chord pairs. The study finds that deep learning models, particularly those incorporating functional harmony, outperform non-deep learning approaches in both objective metrics and subjective evaluations.

ABSTRACT

Several prior works have proposed various methods for the task of automatic melody harmonization, in which a model aims to generate a sequence of chords to serve as the harmonic accompaniment of a given multiple-bar melody sequence. In this paper, we present a comparative study evaluating and comparing the performance of a set of canonical approaches to this task, including a template matching based model, a hidden Markov based model, a genetic algorithm based model, and two deep learning based models. The evaluation is conducted on a dataset of 9,226 melody/chord pairs we newly collect for this study, considering up to 48 triad chords, using a standardized training/test split. We report the result of an objective evaluation using six different metrics and a subjective study with 202 participants.

Motivation & Objective

• To evaluate and compare the performance of canonical melody harmonization methods, including traditional statistical models and deep learning approaches.
• To collect and release a new dataset of 9,226 melody/chord pairs for standardized benchmarking of melody harmonization models.
• To assess model performance using both objective metrics and a large-scale subjective user study with 202 participants.
• To investigate how different model architectures and incorporated music theory (e.g., functional harmony) affect harmonization quality.
• To provide a foundation for future research by decoupling model architecture from music knowledge integration in harmonization tasks.

Proposed method

• The study implements and evaluates five models: template matching, hidden Markov model (HMM), genetic algorithm (GA), bidirectional LSTM (BiLSTM), and a functional harmony-aware deep learning model.
• A standardized 80/20 training/test split is applied to all models using a newly collected dataset of 9,226 symbolic music pairs.
• Six objective metrics are used to evaluate chord progression quality, including harmonicity, voice leading, and chord tone to non-chord tone ratios.
• A subjective evaluation with 202 participants ranks model outputs in pairwise comparisons to assess perceived musical quality and pleasantness.
• The functional harmony-aware model encodes chord functions (e.g., tonic, dominant) as part of the input representation to improve musical coherence.
• All models are evaluated under identical conditions to ensure fair comparison, with chord labels reduced to major/minor triads for consistency.

Experimental results

Research questions

• RQ1 How do traditional statistical models (e.g., HMMs, template matching) compare to modern deep learning models in melody harmonization quality?
• RQ2 To what extent does incorporating functional harmony knowledge improve the performance of deep learning models in melody harmonization?
• RQ3 How do objective metrics correlate with human perception in evaluating harmonization quality?
• RQ4 Can pairwise subjective evaluation reduce bias in human ratings compared to absolute scoring?
• RQ5 What are the relative strengths and limitations of different model architectures when generating musically plausible chord progressions?

Key findings

• Deep learning models, particularly the BiLSTM-based model with functional harmony encoding, achieved the highest performance across all objective metrics and subjective evaluations.
• The functional harmony-aware model outperformed other deep learning and non-deep learning models in both harmonicity and perceived musical interest.
• Subjective ratings showed that models were perceived as more pleasant when paired with human-composed progressions, but the functional harmony model ranked highest overall in pairwise comparisons.
• The template matching and HMM-based models performed reasonably well but were consistently outperformed by deep learning approaches in both objective and subjective assessments.
• The genetic algorithm-based model showed moderate performance, indicating that evolutionary search strategies can generate plausible progressions but lack the generalization of neural networks.
• The study confirms that incorporating music-theoretic knowledge (e.g., functional harmony) significantly enhances model performance, especially in generating musically coherent progressions.