[Paper Review] Meta-song evaluation for chord recognition

Yizhao Ni, Matt McVicar+2 more|arXiv (Cornell University)|Sep 2, 2011

Music and Audio Processing11 references3 citations

TL;DR

This paper proposes meta-song evaluation, a scalable method to estimate chord recognition system performance on unannotated songs by generating high-accuracy pseudo-annotations from online chord databases and modeling the relationship between pseudo-accuracy and ground-truth accuracy via statistical frameworks. The approach enables reliable generalization analysis across genres and outperforms existing metrics, with consensus modeling improving performance by compensating for system-specific weaknesses.

ABSTRACT

We present a new approach to evaluate chord recognition systems on songs which do not have full annotations. The principle is to use online chord databases to generate high accurate "pseudo annotations" for these songs and compute "pseudo accuracies" of test systems. Statistical models that model the relationship between "pseudo accuracy" and real performance are then applied to estimate test systems' performance. The approach goes beyond the existing evaluation metrics, allowing us to carry out extensive analysis on chord recognition systems, such as their generalizations to different genres. In the experiments we applied this method to evaluate three state-of-the-art chord recognition systems, of which the results verified its reliability.

Motivation & Objective

• To address the limitation of existing chord recognition evaluation, which relies on small, genre-biased datasets with full ground-truth annotations.
• To enable performance estimation of chord recognition systems on large, unannotated song collections lacking ground-truth labels.
• To analyze system generalization across diverse musical genres beyond the constraints of current benchmark datasets.
• To develop a scalable, cost-effective alternative to manual annotation for evaluating chord recognition systems.
• To explore ensemble methods that improve recognition accuracy by combining predictions from multiple systems.

Proposed method

• Generate high-accuracy pseudo-annotations for unannotated songs using online chord databases (e.g., E-chords) and a reference system (Jump Alignment) to correct timing and errors.
• Use a validation set with both ground-truth and pseudo-accuracies to model the relationship between pseudo-accuracy and real performance via three statistical frameworks: single Gaussian, individual Gaussian, and linear regression.
• Estimate ground-truth accuracies on a large test set (1,840 songs) using the learned models, with confidence intervals to assess reliability.
• Apply the models to estimate system performance per genre, enabling cross-genre generalization analysis.
• Combine predictions from three systems (HP, labROSA, Chordino) via majority voting to create a consensus prediction system.
• Validate model reliability by testing on the validation set and confirming that all real ground-truth accuracies fall within 95% confidence intervals.

Experimental results

Research questions

• RQ1 Can pseudo-accuracies derived from online chord databases reliably estimate real ground-truth performance of chord recognition systems?
• RQ2 How do different statistical models (single Gaussian, individual Gaussian, linear regression) compare in estimating ground-truth accuracy from pseudo-annotations?
• RQ3 To what extent can chord recognition systems generalize across different musical genres, and how does performance vary per genre?
• RQ4 Can combining predictions from multiple systems through consensus improve overall recognition accuracy?
• RQ5 How do outliers in online chord sequences affect model reliability, and can they be detected and mitigated?

Key findings

• The proposed statistical models (S, I, L) reliably estimated ground-truth accuracies, with all real GT accuracies falling within the 95% confidence intervals when validated on the test set.
• The HP system outperformed labROSA and Chordino on most genres, especially Rock-related genres, consistent with its training data distribution.
• The consensus system, formed by majority voting, achieved slightly better performance than HP alone, particularly compensating for HP’s weaknesses in genres like Funk and Blues.
• The single Gaussian model showed a bias toward HP due to shared chromagram features, which was corrected by the individual Gaussian and linear regression models.
• Performance estimates across genres revealed that labROSA and Chordino performed similarly, with minor variations likely due to low song counts in some genres.
• Outliers in online chord sequences—such as those with incomplete or solo-only annotations—were identified as sources of poor correlation between pseudo- and ground-truth accuracies, suggesting a need for future outlier detection.