[Paper Review] InverseMV: Composing Piano Scores with a Convolutional Video-Music Transformer

Chin‐Teng Lin, Yang Mu|arXiv (Cornell University)|Dec 31, 2021

Music and Audio Processing4 citations

TL;DR

This paper proposes VMT, a video-music transformer model that generates symbolic piano scores from video frames using multimodal attention mechanisms. It introduces a new dataset of 7+ hours of video-MIDI aligned pop music videos and demonstrates through human evaluation that VMT outperforms a Seq2seq baseline in music smoothness and video relevance, especially in capturing frame-level motion and rhythmic structure.

ABSTRACT

Many social media users prefer consuming content in the form of videos rather than text. However, in order for content creators to produce videos with a high click-through rate, much editing is needed to match the footage to the music. This posts additional challenges for more amateur video makers. Therefore, we propose a novel attention-based model VMT (Video-Music Transformer) that automatically generates piano scores from video frames. Using music generated from models also prevent potential copyright infringements that often come with using existing music. To the best of our knowledge, there is no work besides the proposed VMT that aims to compose music for video. Additionally, there lacks a dataset with aligned video and symbolic music. We release a new dataset composed of over 7 hours of piano scores with fine alignment between pop music videos and MIDI files. We conduct experiments with human evaluation on VMT, SeqSeq model (our baseline), and the original piano version soundtrack. VMT achieves consistent improvements over the baseline on music smoothness and video relevance. In particular, with the relevance scores and our case study, our model has shown the capability of multimodality on frame-level actors' movement for music generation. Our VMT model, along with the new dataset, presents a promising research direction toward composing the matching soundtrack for videos. We have released our code at https://github.com/linchintung/VMT

Motivation & Objective

• To address the lack of datasets and models for video-to-music generation, especially for symbolic piano music.
• To develop a multimodal model that can generate music that aligns with video content at the frame level.
• To overcome copyright issues associated with using pre-existing music in videos.
• To improve music generation quality by modeling long-term structure and rhythmic coherence in symbolic music.
• To establish a benchmark for video-music alignment through human evaluation and case studies.

Proposed method

• Proposes a Video-Music Transformer (VMT) model that uses self-attention and cross-attention mechanisms to process video frames and generate symbolic piano music in MIDI format.
• Adapts a convolutional video encoder to extract spatiotemporal features from input video frames at frame-level granularity.
• Uses a causal autoregressive transformer decoder to generate sequential MIDI tokens with autoregressive generation, maintaining long-term musical structure.
• Leverages a pre-trained Music Transformer backbone for better modeling of musical sequences and harmonic progression.
• Trains the model end-to-end using cross-entropy loss on token-level MIDI note predictions, including pitch, duration, and velocity.
• Introduces a novel data pipeline to align video frames with precise MIDI note on/off events, ensuring fine-grained synchronization.

Experimental results

Research questions

• RQ1 Can a multimodal transformer model generate piano scores that are perceptually smooth and musically coherent when conditioned on video frames?
• RQ2 How well can the model align generated music with frame-level visual dynamics such as dancer movements or scene transitions?
• RQ3 Does the proposed model outperform a strong Seq2Seq baseline in both music quality and video relevance?
• RQ4 To what extent can the model learn and reproduce recurring melodic motifs and rhythmic patterns from video input?
• RQ5 Can the model generalize to diverse pop music videos with varying camera angles, lighting, and performance styles?

Key findings

• VMT significantly outperforms the Seq2Seq baseline in music smoothness, with human evaluators consistently rating its output as more coherent and musically natural.
• VMT generates music with recurring melodic motifs and varied pitch contours, demonstrating the ability to capture harmonic and rhythmic structure similar to human-composed music.
• Human evaluation shows VMT achieves higher video relevance scores, particularly in matching music to frame-level actions such as dancing or gestures.
• The model successfully generates music with consistent note lengths and proper on/off event alignment, avoiding the repetitive note patterns common in the Seq2Seq baseline.
• Case studies confirm that VMT can detect and respond to visual cues like dancer motion, producing music that reflects the rhythm and emotion of the video.
• The release of the new dataset—over 7 hours of video-MIDI aligned pop music videos—provides a valuable benchmark for future video-to-music generation research.