[Paper Review] Learning Complex Basis Functions for Invariant Representations of Audio

Stefan Lattner, Monika Dörfler+1 more|arXiv (Cornell University)|Jul 13, 2019

Music and Audio Processing35 references4 citations

TL;DR

This paper proposes the Complex Autoencoder (CAE), a deep learning architecture that learns complex basis functions from audio spectrograms to produce transformation-invariant representations. By projecting audio into a magnitude space invariant to orthogonal transformations (e.g., transposition, time-shift) and a phase space encoding transformation differences, the CAE achieves state-of-the-art performance in audio-to-score alignment and repeated section discovery, outperforming Chroma and GAE-based features.

ABSTRACT

Learning features from data has shown to be more successful than using hand-crafted features for many machine learning tasks. In music information retrieval (MIR), features learned from windowed spectrograms are highly variant to transformations like transposition or time-shift. Such variances are undesirable when they are irrelevant for the respective MIR task. We propose an architecture called Complex Autoencoder (CAE) which learns features invariant to orthogonal transformations. Mapping signals onto complex basis functions learned by the CAE results in a transformation-invariant "magnitude space" and a transformation-variant "phase space". The phase space is useful to infer transformations between data pairs. When exploiting the invariance-property of the magnitude space, we achieve state-of-the-art results in audio-to-score alignment and repeated section discovery for audio. A PyTorch implementation of the CAE, including the repeated section discovery method, is available online.

Motivation & Objective

• To address the limitation of hand-crafted audio features being sensitive to irrelevant transformations like transposition and time-shift in music information retrieval (MIR).
• To develop a method that learns basis functions from data rather than relying on fixed transforms like Fourier or CQT.
• To produce a magnitude space invariant to orthogonal transformations and a phase space that encodes transformation types and distances.
• To improve performance on MIR tasks such as audio-to-score alignment and repeated section discovery through invariant representations.
• To demonstrate the model’s generalization to other invariances, such as rotation, using MNIST as a benchmark.

Proposed method

• The CAE learns complex basis functions from constant-Q transformed (CQT) audio representations through an autoencoder architecture with complex-valued weights and activations.
• The model is trained on data pairs related by orthogonal transformations (e.g., transposition, time-shift), using a reconstruction loss in the complex domain.
• The magnitude of the encoded representation forms a transformation-invariant space, while the phase encodes transformation-specific information.
• The magnitude space is used for downstream tasks requiring invariance, while the phase space enables inference of transformation types and distances between signals.
• The architecture is trained in an unsupervised manner, with no need for explicit labels on transformations.
• The method generalizes to other invariances, such as rotation, as demonstrated on the MNIST dataset with rotated digits.

Experimental results

Research questions

• RQ1 Can learning complex basis functions from audio data produce more robust, transformation-invariant representations than fixed transforms like CQT or Chroma?
• RQ2 Does the magnitude space of the CAE achieve better invariance to transposition and time-shift than existing methods in audio-to-score alignment?
• RQ3 Can the phase space of the CAE effectively encode transformation types and distances between audio fragments, enabling better structure analysis?
• RQ4 How does the CAE compare to GAE-based features in terms of robustness to tempo variations in alignment tasks?
• RQ5 Can the CAE generalize to other invariances, such as rotation, in non-audio domains like image classification?

Key findings

• The CAE achieved state-of-the-art performance in transposition-invariant repeated section discovery, outperforming Chroma and GAE-based features.
• In audio-to-score alignment, CAE features yielded better results than both Chroma and GAE-based features, particularly under tempo variations.
• On the rotated MNIST dataset, logistic regression on the CAE’s magnitude space achieved significantly lower error rates than on the input space, even with only 256 learned basis functions.
• The CAE’s magnitude space formed clear, class-separated clusters in PCA visualization, indicating strong invariance to rotation.
• The phase-difference space of the CAE clearly represented transformation types (e.g., rotation angles), enabling discriminative analysis of signal relations.
• The CAE outperformed a GAE with 1000 basis functions using only 256, suggesting more efficient and effective basis learning through explicit training for invariance.