[Paper Review] SingSong: Generating musical accompaniments from singing

Chris Donahue, Antoine Caillon+9 more|arXiv (Cornell University)|Jan 30, 2023

Speech and Audio ProcessingComputer Science30 references8 citations

TL;DR

SingSong generates instrumental music to accompany input vocals by training a conditional audio-gen model on source-separated vocal-instrumental pairs and adapting AudioLM for audio-to-audio generation. It improves generalization to isolated vocals and is preferred by listeners over a strong retrieval baseline.

ABSTRACT

We present SingSong, a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Specifically, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022) -- a state-of-the-art approach for unconditional audio generation -- to be suitable for conditional "audio-to-audio" generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a significant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline. Sound examples at https://g.co/magenta/singsong

Motivation & Objective

• Motivate intuitive music creation by letting people generate accompaniment with their own singing.
• Leverage state-of-the-art source separation to create paired vocal-instrumental data for training.
• Adapt an unconditional audio generation model (AudioLM) to a conditional, audio-to-audio setting.
• Address generalization from source-separated training data to real-world isolated vocals.
• Evaluate perceptual quality and provide guidance for improving fidelity and coherence of generated accompaniment.

Proposed method

• Create a large paired dataset by applying an off-the-shelf source separation algorithm to 1 million tracks to obtain aligned vocal and instrumental sources.
• Adapt AudioLM into a conditional audio-to-audio framework to generate instrumentals given vocals, trained on the source-separated pairs.
• Featurize inputs by exploring vocal representations (semantic and coarse acoustic codes) and introduce noise to conceal source-artifacts for better generalization.
• Train a sequence-to-sequence model (T5-based encoder-decoder) to map vocal features to instrumental semantic and coarse acoustic codes, followed by decoding into waveform via SoundStream.
• Use a multi-stage generation process: sample semantic and coarse codes conditioned on vocal features, then refine with coarse-to-fine acoustic codes before waveform reconstruction; mix generated instrumental with input vocals.
• Experiment with featurization strategies (Noisy, SA-SA, S-SA, etc.) and model scales (Base vs XL) to optimize generalization to isolated vocals.

Experimental results

Research questions

• RQ1 Can a generative model be conditioned on vocal inputs to produce coherent instrumental accompaniment in real time?
• RQ2 Does using source-separated data for training enable effective audio-to-audio accompaniment generation for real-world isolated vocals?
• RQ3 What vocal conditioning featurizations best generalize from source-separated training data to isolated vocal inputs?
• RQ4 How do model scale and featurization choices affect perceptual quality and generalization gap in accompaniment generation?
• RQ5 How does SingSong compare to retrieval-based baselines in human listener judgments of musical compatibility?

Key findings

• Listeners significantly preferred SingSong instrumentals over a strong retrieval baseline for the same vocals.
• Best-performing configurations (Noisy / S-SA) substantially improve generalization to isolated vocals (55% relative improvement in FAD_i over a naive AudioLM adaptation).
• Scaling to a larger model (SingSong-XL) yields perceptual improvements over the base model in listening tests.
• Removing vocal coarse acoustic codes from conditioning (S-SA) or adding noise to vocals improves generalization and stabilizes training.
• Instrumentals tend to have stronger percussive/content coherence than harmonic elements, indicating room for improving harmonic context in conditioning.