[Paper Review] Generation of lyrics lines conditioned on music audio clips

Olga Vechtomova, Gaurav Sahu+1 more|arXiv (Cornell University)|Sep 30, 2020

Music and Audio Processing14 references4 citations

TL;DR

This paper presents a bimodal variational autoencoder model that generates emotionally congruent lyrics lines conditioned on instrumental music audio clips. By learning continuous latent representations from spectrograms and using them to condition a text VAE, the system produces lyrics that align with the emotional tone of the input audio, as validated by both automatic metrics and human evaluation showing 78.3% accuracy in detecting audio-lyrics consistency.

ABSTRACT

We present a system for generating novel lyrics lines conditioned on music audio. A bimodal neural network model learns to generate lines conditioned on any given short audio clip. The model consists of a spectrogram variational autoencoder (VAE) and a text VAE. Both automatic and human evaluations demonstrate effectiveness of our model in generating lines that have an emotional impact matching a given audio clip. The system is intended to serve as a creativity tool for songwriters.

Motivation & Objective

• To develop a creative tool that generates novel, emotionally resonant lyrics lines for songwriters based on instrumental audio clips.
• To address the gap in existing text generation models that do not condition on music audio, which plays a key role in emotional impact.
• To explore whether continuous latent variables from a spectrogram-VAE can effectively condition a text-VAE for style-aligned generation.
• To evaluate the model’s ability to generate lyrics that match the emotional tone of diverse instrumental music clips.

Proposed method

• A spectrogram-VAE learns a continuous latent representation of 10-second music audio clips by encoding MEL spectrograms into a latent space.
• The text-VAE decoder is conditioned on the latent vector from the spectrogram-VAE, enabling generation of lyrics aligned with the audio’s emotional and stylistic features.
• The model uses LSTMs for both encoder and decoder networks in the VAEs, with reconstruction loss and KL divergence regularization.
• Word-level KL divergence and rank-biased overlap (RBO) are used to measure lexical similarity across generated lyrics, assessing consistency within emotional categories.
• Human evaluation involves participants identifying which of two lyric lists (from same or opposite emotional category) matches a given instrumental clip.
• The system is trained on a dataset of instrumental clips from the 'Ghosts I-IV' album, split into 'calm' and 'intense' emotional categories.

Experimental results

Research questions

• RQ1 Can a neural model generate lyrics lines that are emotionally congruent with a given instrumental music audio clip?
• RQ2 Can continuous latent representations learned from spectrograms effectively condition a text generation model to produce stylistically and emotionally aligned lyrics?
• RQ3 Do generated lyrics show greater lexical similarity within emotional categories (e.g., 'calm' vs. 'intense') than across categories?
• RQ4 To what extent can human evaluators detect the emotional alignment between generated lyrics and the conditioning audio clip?

Key findings

• The model successfully generates lyrics lines that are emotionally aligned with the input audio, as demonstrated by a human evaluation achieving 78.3% accuracy in identifying the correct audio-lyrics pairing.
• Lines generated from 'calm' music clips showed higher RBO scores with each other (mean RBO = 0.45) than with lines from 'intense' clips, indicating greater lexical consistency within the calm category.
• Lines from 'intense' clips showed lower RBO values among themselves (mean RBO = 0.32), suggesting greater lexical diversity, likely due to more varied musical features in the training data.
• KL divergence analysis confirmed that word distributions in generated lyrics differ meaningfully between 'calm' and 'intense' categories, with statistically significant dB differences (p < 0.05).
• The spectrogram-VAE's continuous latent space effectively encodes musical characteristics that influence lyrical style and emotion, enabling meaningful conditioning of the text-VAE.
• The model demonstrates that audio-derived continuous embeddings can be used to condition text generation, extending prior work that relied on discrete text-based style embeddings.