[Paper Review] Adversarial Learning for Improved Onsets and Frames Music Transcription

Jong Wook Kim, Juan Pablo Bello|arXiv (Cornell University)|Jun 20, 2019

Music and Audio Processing45 references21 citations

TL;DR

This paper proposes an adversarial training framework that improves music transcription by modeling inter-label dependencies in time-frequency representations, using a conditional GAN to refine predictions from the Onsets and Frames model. The method significantly boosts frame and note F1 scores—achieving p < 10⁻¹⁴ significance—by producing sharper, more confident outputs that reduce error rates and improve generalization.

ABSTRACT

Automatic music transcription is considered to be one of the hardest problems in music information retrieval, yet recent deep learning approaches have achieved substantial improvements on transcription performance. These approaches commonly employ supervised learning models that predict various time-frequency representations, by minimizing element-wise losses such as the cross entropy function. However, applying the loss in this manner assumes conditional independence of each label given the input, and thus cannot accurately express inter-label dependencies. To address this issue, we introduce an adversarial training scheme that operates directly on the time-frequency representations and makes the output distribution closer to the ground-truth. Through adversarial learning, we achieve a consistent improvement in both frame-level and note-level metrics over Onsets and Frames, a state-of-the-art music transcription model. Our results show that adversarial learning can significantly reduce the error rate while increasing the confidence of the model estimations. Our approach is generic and applicable to any transcription model based on multi-label predictions, which are very common in music signal analysis.

Motivation & Objective

• Address the limitation of standard supervised learning in music transcription, which assumes conditional independence between labels and fails to model inter-label dependencies in time-frequency representations.
• Improve the performance of state-of-the-art models like Onsets and Frames by incorporating structured prediction through adversarial training.
• Reduce transcription errors and increase prediction confidence by learning a more accurate joint distribution over note activations.
• Demonstrate that adversarial training acts as a regularizer, improving generalization and reducing overfitting on unseen data.
• Provide a generic, plug-in solution applicable to any model predicting multi-label time-frequency representations in music signal analysis.

Proposed method

• Introduce a conditional generative adversarial network (cGAN) where the transcription model acts as a conditional generator, and a discriminator is trained to distinguish real ground-truth representations from generated predictions.
• Apply an adversarial loss that encourages the generator to produce output distributions closer to the real data distribution, thereby modeling complex inter-label dependencies along both time and frequency axes.
• Use a least-squares GAN or non-saturating GAN loss to stabilize training and improve the quality of generated posteriorgrams.
• Train the generator and discriminator in a competitive manner, with the discriminator providing a learned prior that favors realistic, high-confidence note sequences.
• Integrate the adversarial loss with the standard element-wise cross-entropy loss in a multi-objective training scheme to preserve accuracy while enhancing structural consistency.
• Apply mixup augmentation during training to further improve robustness, with optimal performance observed at α = 0.3.

Experimental results

Research questions

• RQ1 Can adversarial training effectively model inter-label dependencies in two-dimensional music transcription outputs, beyond what standard element-wise losses can achieve?
• RQ2 Does incorporating a GAN-based regularizer improve generalization and reduce overfitting in music transcription models trained on limited data?
• RQ3 To what extent does adversarial learning enhance prediction confidence and reduce blurry outputs in frame-level and note-level transcription metrics?
• RQ4 How does the proposed method compare to strong baselines like Onsets and Frames in terms of F1 score, recall, and precision across multiple evaluation metrics?
• RQ5 Is the adversarial framework generic and transferable to other models that predict multi-label time-frequency representations in music and audio processing?

Key findings

• The proposed adversarial method achieves statistically significant improvements in both frame F1 (up to 0.914) and note F1 (up to 0.956) on the MAESTRO test set, with p < 10⁻¹⁴ for all metrics.
• The model produces sharper, more confident posteriorgrams with fewer values in the ambiguous range (0.1–0.9), reducing sensitivity to thresholding in post-processing.
• The generalization gap between training and validation F1 scores is significantly reduced, indicating that the GAN loss acts as an effective regularizer.
• The non-saturating GAN loss with mixup strength α = 0.3 yields the best performance, outperforming both baseline and other GAN variants.
• Improvements are consistently distributed across tracks, suggesting robustness and broad applicability rather than localized gains on specific pieces.
• The method is generic and applicable to any model relying on multi-label time-frequency predictions, including those in speech and music synthesis.