[Paper Review] Learning Soft-Attention Models for Tempo-invariant Audio-Sheet Music Retrieval

Stefan Balke, Matthias Dorfer+3 more|arXiv (Cornell University)|Jun 26, 2019

Music and Audio Processing27 references4 citations

TL;DR

This paper proposes a soft-attention mechanism for audio spectrograms in end-to-end audio-sheet music retrieval, enabling the model to dynamically adjust its temporal context based on tempo and note density. The attention mechanism improves retrieval robustness across varying tempos, achieving state-of-the-art performance on synthesized piano data by focusing on relevant audio regions while maintaining interpretability.

ABSTRACT

Connecting large libraries of digitized audio recordings to their corresponding sheet music images has long been a motivation for researchers to develop new cross-modal retrieval systems. In recent years, retrieval systems based on embedding space learning with deep neural networks got a step closer to fulfilling this vision. However, global and local tempo deviations in the music recordings still require careful tuning of the amount of temporal context given to the system. In this paper, we address this problem by introducing an additional soft-attention mechanism on the audio input. Quantitative and qualitative results on synthesized piano data indicate that this attention increases the robustness of the retrieval system by focusing on different parts of the input representation based on the tempo of the audio. Encouraged by these results, we argue for the potential of attention models as a very general tool for many MIR tasks.

Motivation & Objective

• To address the challenge of fixed temporal context in audio embeddings for cross-modal retrieval, which causes performance degradation under tempo variations.
• To improve retrieval robustness in audio-sheet music matching by allowing the model to adaptively focus on relevant audio segments based on tempo and note density.
• To demonstrate that soft-attention mechanisms can serve as a general-purpose solution for tempo-invariant retrieval in music information retrieval (MIR).
• To validate the intuitive and interpretable behavior of the attention mechanism through qualitative and quantitative analysis on synthesized piano data.

Proposed method

• A dual-branch convolutional neural network (CNN) architecture is used, with separate pathways for audio spectrograms and sheet music images, both projecting into a shared 32-dimensional embedding space.
• A soft-attention mechanism is applied to the audio spectrogram input, learning to assign variable weights across time frames based on content relevance.
• The attention mechanism is trained end-to-end with a contrastive loss function to optimize embedding similarity between matching audio and sheet music pairs.
• The attention weights are computed using a learnable attention network that processes the raw audio spectrogram and outputs a soft, differentiable distribution over time frames.
• The model uses canonical correlation analysis (CCA) to align the audio and sheet music embeddings in the shared space, enforcing semantic correspondence.
• The system is evaluated on a synthesized piano dataset with controlled tempo variations, using cosine distance for nearest-neighbor retrieval.

Experimental results

Research questions

• RQ1 Can a soft-attention mechanism improve retrieval performance in audio-sheet music matching under varying playback tempos?
• RQ2 How does the attention mechanism adapt its focus based on note density and tempo in the audio spectrogram?
• RQ3 Does the attention mechanism produce interpretable and intuitive attention distributions that correlate with musical content?
• RQ4 To what extent does the attention mechanism reduce sensitivity to fixed-size audio input windows in cross-modal retrieval?

Key findings

• The proposed soft-attention mechanism significantly improves retrieval performance over the baseline model without attention, particularly under tempo variations.
• The attention mechanism adapts its focus: it becomes more concentrated (low entropy) when note density is high and more diffuse (high entropy) when note density is low, as confirmed by entropy vs. onset count analysis.
• Qualitative examples show that the attention network correctly identifies relevant musical regions, such as melodic passages or rhythmic clusters, even under extreme tempo changes.
• The model maintains high retrieval accuracy across a wide range of tempos, from 78 to 250 bpm, demonstrating strong tempo invariance.
• The attention mechanism effectively handles pieces with long sustained chords by distributing attention across the entire audio window, ensuring full content matching.
• The results suggest that attention mechanisms can serve as a general-purpose tool for improving robustness in cross-modal MIR tasks beyond audio-sheet music retrieval.