[Paper Review] Foley Music: Learning to Generate Music from Videos

Chuang Gan, Deng Huang+3 more|arXiv (Cornell University)|Jul 21, 2020

Music and Audio Processing66 references10 citations

TL;DR

This paper introduces Foley Music, a system that generates expressive, synchronized music from silent videos of musicians by modeling body keypoints and MIDI events as intermediate representations. Using a Graph-Transformer framework that maps human motion to symbolic music, the model outperforms baselines in listener studies and enables flexible music editing through interpretable MIDI outputs.

ABSTRACT

In this paper, we introduce Foley Music, a system that can synthesize plausible music for a silent video clip about people playing musical instruments. We first identify two key intermediate representations for a successful video to music generator: body keypoints from videos and MIDI events from audio recordings. We then formulate music generation from videos as a motion-to-MIDI translation problem. We present a Graph$-$Transformer framework that can accurately predict MIDI event sequences in accordance with the body movements. The MIDI event can then be converted to realistic music using an off-the-shelf music synthesizer tool. We demonstrate the effectiveness of our models on videos containing a variety of music performances. Experimental results show that our model outperforms several existing systems in generating music that is pleasant to listen to. More importantly, the MIDI representations are fully interpretable and transparent, thus enabling us to perform music editing flexibly. We encourage the readers to watch the demo video with audio turned on to experience the results.

Motivation & Objective

• To develop a system that generates plausible, synchronized music from silent videos of musical performances.
• To address the challenge of mapping visual motion to expressive audio by identifying key intermediate representations: body keypoints and MIDI events.
• To enable flexible music editing by using fully interpretable MIDI representations instead of raw waveforms.
• To demonstrate that motion-to-MIDI translation via a Graph-Transformer model yields higher-quality, more natural-sounding music than prior approaches.

Proposed method

• The system detects 2D body and hand keypoints from video frames using pose estimation to represent human motion over time.
• It formulates music generation as a motion-to-MIDI translation problem, using MIDI as a symbolic, interpretable audio representation encoding note-on/off, pitch, and velocity.
• A Graph Convolutional Network (GCN) encoder processes the temporal sequence of keypoint coordinates to learn spatial-temporal features.
• A Transformer decoder with self-attention captures long-range dependencies in the music generation process, predicting MIDI event sequences.
• The predicted MIDI sequences are converted into realistic audio using a standard synthesizer for evaluation and playback.
• The model is trained on large-scale music performance videos across nine instruments, including piano, violin, and accordion.

Experimental results

Research questions

• RQ1 Can a video-to-music generation system accurately predict expressive music based solely on visual motion cues from silent videos?
• RQ2 Is the use of body keypoints and MIDI as intermediate representations more effective than raw visual or audio features for cross-modal music generation?
• RQ3 Can the symbolic nature of MIDI enable flexible, interpretable music editing that is not feasible with waveform-based models?
• RQ4 Does a Graph-Transformer architecture outperform recurrent or convolutional models in capturing long-term musical dependencies from motion sequences?

Key findings

• The proposed Graph-Transformer model significantly outperformed strong baselines in listener studies, with higher ratings for correctness, synchronization, and overall preference.
• The model achieved a negative log-likelihood (NLL) loss of 1.558 on violin and 1.995 on flute in MIDI prediction, outperforming RGB image and optical flow baselines.
• The ablation study confirmed that keypoint-based visual representations yield better MIDI prediction accuracy than RGB or optical flow features.
• The use of a Transformer decoder reduced NLL loss compared to GRU, demonstrating its superiority in modeling long-range musical dependencies.
• The system generated music with lower NDB (20) than all baselines, indicating higher diversity and better generalization.
• MIDI representations enabled direct editing of musical key and style, demonstrating unprecedented flexibility in controllable music generation.