[Paper Review] Cross-modal variational inference for bijective signal-symbol translation

Axel Chemla--Romeu-Santos, Stavros Ntalampiras+3 more|arXiv (Cornell University)|Feb 10, 2020

Music and Audio Processing4 citations

TL;DR

This paper proposes a bijective, variational inference-based framework for cross-modal signal-symbol translation using two domain-specific variational autoencoders (VAEs) with shared latent spaces. By enforcing additive matching between latent representations, the model enables bidirectional generation—audio-to-symbol transcription and symbol-to-audio synthesis—while supporting arbitrary symbolic vocabularies and creative control via continuous latent space exploration.

ABSTRACT

Extraction of symbolic information from signals is an active field of research enabling numerous applications especially in the Musical Information Retrieval domain. This complex task, that is also related to other topics such as pitch extraction or instrument recognition, is a demanding subject that gave birth to numerous approaches, mostly based on advanced signal processing-based algorithms. However, these techniques are often non-generic, allowing the extraction of definite physical properties of the signal (pitch, octave), but not allowing arbitrary vocabularies or more general annotations. On top of that, these techniques are one-sided, meaning that they can extract symbolic data from an audio signal, but cannot perform the reverse process and make symbol-to-signal generation. In this paper, we propose an bijective approach for signal/symbol translation by turning this problem into a density estimation task over signal and symbolic domains, considered both as related random variables. We estimate this joint distribution with two different variational auto-encoders, one for each domain, whose inner representations are forced to match with an additive constraint, allowing both models to learn and generate separately while allowing signal-to-symbol and symbol-to-signal inference. In this article, we test our models on pitch, octave and dynamics symbols, which comprise a fundamental step towards music transcription and label-constrained audio generation. In addition to its versatility, this system is rather light during training and generation while allowing several interesting creative uses that we outline at the end of the article.

Motivation & Objective

• Address the limitations of one-sided, non-generic signal-to-symbol transcription methods in music information retrieval.
• Enable bidirectional translation between audio signals and symbolic annotations (e.g., pitch, dynamics, octave) using a unified generative model.
• Support arbitrary symbolic vocabularies, including complex musical symbols like trills or playing modes, beyond basic pitch detection.
• Provide a unified latent space for both signal and symbol generation, enabling creative audio synthesis and control.
• Integrate source separation with explicit decomposition to improve instrument recognition and model interpretability.

Proposed method

• Formulate signal-symbol translation as a joint density estimation problem over audio and symbolic domains.
• Train two separate VAEs—one for spectral features (signal) and one for symbolic annotations (e.g., pitch, dynamics)—with shared latent spaces.
• Enforce latent space alignment via an additive constraint between the two VAEs’ latent representations to enable cross-domain inference.
• Use variational inference to approximate posterior distributions over latent variables for both signal and symbol domains.
• Enable cross-domain generation by decoding a latent code from one domain using the decoder of the other domain (e.g., signal encoder → symbol decoder).
• Integrate a novel source-separation module using disjoint decoders and a Bayesian inference network over mixture coefficients to improve instrument recognition.

Experimental results

Research questions

• RQ1 Can a unified variational autoencoder framework achieve bidirectional, bijective translation between audio signals and symbolic music annotations?
• RQ2 How well can such a model generalize to arbitrary symbolic vocabularies beyond standard pitch and dynamics?
• RQ3 To what extent does latent space sharing enable controllable audio synthesis and symbolic generation?
• RQ4 Can the model support creative audio generation tasks such as spectral morphing and sequence-based navigation in latent space?
• RQ5 How does the integration of explicit source separation improve performance in polyphonic music transcription?

Key findings

• The model achieves bidirectional translation: it performs accurate signal-to-symbol transcription and symbol-to-signal generation using a shared, structured latent space.
• The system supports arbitrary symbolic vocabularies, including complex musical symbols like trills or articulations, enabling flexible and user-defined annotation schemes.
• Latent space exploration enables diverse creative applications such as spectral morphing, free trajectory navigation, and sequence-based audio generation.
• The model supports semi-supervised generation by conditioning on symbolic labels (e.g., specific pitch, dynamics) and sampling diverse audio realizations.
• The integration of source separation with disjoint decoders improves instrument recognition and enables more accurate decomposition of polyphonic mixtures.
• The approach is computationally efficient during both training and inference, enabling real-time and interactive applications.