[Paper Review] Equivariant Self-Supervision for Musical Tempo Estimation

Elio Quinton|arXiv (Cornell University)|Sep 3, 2022

Music and Audio Processing4 citations

TL;DR

This paper proposes a novel self-supervised learning method for musical tempo estimation that uses equivariance to audio time-stretching as a pretext task, eliminating the need for labeled data, negative sampling, or regularization. It achieves performance comparable to supervised methods on multiple benchmarks using only unlabelled audio and moderate computational resources.

ABSTRACT

Self-supervised methods have emerged as a promising avenue for representation learning in the recent years since they alleviate the need for labeled datasets, which are scarce and expensive to acquire. Contrastive methods are a popular choice for self-supervision in the audio domain, and typically provide a learning signal by forcing the model to be invariant to some transformations of the input. These methods, however, require measures such as negative sampling or some form of regularisation to be taken to prevent the model from collapsing on trivial solutions. In this work, instead of invariance, we propose to use equivariance as a self-supervision signal to learn audio tempo representations from unlabelled data. We derive a simple loss function that prevents the network from collapsing on a trivial solution during training, without requiring any form of regularisation or negative sampling. Our experiments show that it is possible to learn meaningful representations for tempo estimation by solely relying on equivariant self-supervision, achieving performance comparable with supervised methods on several benchmarks. As an added benefit, our method only requires moderate compute resources and therefore remains accessible to a wide research community.

Motivation & Objective

• To address the scarcity and high cost of manually annotated music data in tempo estimation.
• To explore self-supervision in music representation learning beyond supervised or contrastive methods.
• To develop a self-supervision framework that explicitly captures rhythmic structure through equivariance.
• To eliminate reliance on negative sampling or regularization to prevent model collapse.
• To demonstrate that equivariant self-supervision can yield robust, transferable representations for tempo estimation.

Proposed method

• A Siamese network architecture is used, where two views of each audio sample are created via random time-stretching with known factors.
• The model is trained to produce representations that transform predictably under time-stretching, enforcing equivariance rather than invariance.
• A novel loss function is derived that enforces the expected relationship between representations of time-stretched versions, preventing trivial solutions without regularization.
• Audio augmentations (e.g. pitch shift, masking) are applied to improve robustness to non-tempo-related signal variations.
• Pre-training is followed by linear probe fine-tuning on downstream tempo estimation tasks, treating tempo as a multiclass classification problem.
• The method is evaluated on multiple benchmarks (GTZAN, ACM Mirum, Hainsworth, Giantsteps) with zero-shot and fine-tuning protocols.

Experimental results

Research questions

• RQ1 Can equivariant self-supervision learn meaningful tempo representations without any labeled data?
• RQ2 Can a simple, regularisation-free loss function prevent model collapse in self-supervised learning for audio?
• RQ3 Does equivariance-based pre-training yield representations competitive with supervised methods for tempo estimation?
• RQ4 How robust are the learned representations to domain shift and hyperparameter choices?
• RQ5 Can this approach generalize to low-resource music genres or other rhythmic MIR tasks?

Key findings

• The proposed method achieves performance comparable to supervised baselines on all tested datasets, with Accuracy 1 outperforming all supervised methods on GTZAN.
• On the GTZAN dataset, the method achieves 0.958 Accuracy 2, matching the best supervised baseline.
• Performance remains strong across datasets, with Accuracy 1 scores of 0.846 (ACM Mirum), 0.700 (Hainsworth), and 0.952 (Giantsteps) under optimal fine-tuning settings.
• The method shows robustness to domain shift, generalizing well to datasets unseen during pre-training.
• Fine-tuning with time-stretching augmentation improves performance across all datasets, with optimal strength varying per dataset.
• The model achieves competitive results using only a linear head fine-tuning, indicating strong, transferable representations from pre-training.