[Paper Review] A Study of Annotation and Alignment Accuracy for Performance Comparison in Complex Orchestral Music

Thassilo Gadermaier, Gerhard Widmer|arXiv (Cornell University)|Oct 31, 2019

Music and Audio Processing13 references4 citations

TL;DR

This study evaluates human annotation precision and automatic audio-to-audio alignment accuracy for performance comparison in complex orchestral music. By analyzing timing deviations among expert annotators and testing various audio features, it demonstrates that optimized MFCC-based alignment can achieve sub-150ms mean error, approaching human annotation variability, enabling reliable transfer of structural annotations across recordings without full manual re-annotation.

ABSTRACT

Dataset accompanying the paper published at ISMIR 2019. See included README file for details.

Motivation & Objective

• To quantify the timing precision and consistency of human annotations in complex orchestral music performances.
• To establish a ground truth for performance alignment using averaged annotations from expert musicians.
• To evaluate the accuracy of automatic audio-to-audio alignment using diverse audio features and parameters.
• To compare automatic alignment errors with human annotation variability to determine suitability for musicological research.
• To enable efficient transfer of structural annotations (e.g., beat positions) from one annotated recording to others via reliable alignment.

Proposed method

• Collected manual annotations from four expert annotators on three orchestral pieces: Beethoven's Symphony No. 9, Bruckner's Symphony No. 9, and Webern's Symphony Op. 21.
• Used statistical analysis (mean, standard deviation) of time-stamp deviations across annotators to estimate human annotation precision and define ground truth.
• Applied dynamic time warping (DTW) with various audio features—MFCC, modified MFCC, FFT-based features—across different parameter settings (e.g., #MFCC, #skip, FFT size).
• Evaluated alignment performance using distance metrics: L1, L2, cosine, and normalized cross-correlation (cos), reporting mean and maximum errors in milliseconds.
• Compared alignment errors to the standard deviation of human annotations to assess whether automatic alignment is sufficiently accurate for musicological analysis.
• Used a systematic parameter sweep across feature types and configurations to identify optimal settings for alignment accuracy.

Experimental results

Research questions

• RQ1 What is the expected timing precision and consistency of human annotations for score events in complex orchestral music?
• RQ2 How do different audio features and parameter settings affect the accuracy of audio-to-audio alignment in orchestral recordings?
• RQ3 To what extent do automatic alignment errors fall within the range of human annotation variability?
• RQ4 Can automatic alignment be used reliably to transfer structural annotations from one performance to others without manual re-annotation?
• RQ5 Which feature configuration yields the highest alignment accuracy for complex, expressive orchestral performances?

Key findings

• Human annotators exhibit a standard deviation of approximately ±60–80 ms in marking score events, indicating inherent variability in manual timing annotation.
• The best-performing alignment configuration (modified MFCC with 120 coefficients, skipping first 20–40 coefficients, FFT size 4096, cosine distance) achieved a mean error of 69 ms on Webern’s Op. 21-2, within the range of human annotation variability.
• For Bruckner’s Symphony No. 9, 3rd movement, the best alignment achieved a mean error of 116 ms, still within the range of human annotation variation (±60–80 ms), though higher due to complex phrasing.
• Skipping the first 20–40 MFCC coefficients significantly improved alignment accuracy for modified MFCC features, suggesting reduced sensitivity to low-frequency noise.
• No clear relationship was found between FFT size and alignment performance, indicating that feature stability is more dependent on coefficient selection than FFT length.
• Automatic alignment errors were consistently within or slightly above the standard deviation of human annotations, indicating that such methods are suitable for musicological studies requiring comparative performance analysis.