[Paper Review] Audeo: Audio Generation for a Silent Performance Video

Kun Su, Xiulong Liu+1 more|arXiv (Cornell University)|Jun 23, 2020

Music and Audio Processing44 references30 citations

TL;DR

Audeo presents a full three-stage pipeline that converts top-view piano performance videos into synthesized music by predicting a Piano-Roll from video, refining it to a Pseudo-MIDI via a GAN, and then synthesizing audio with traditional or deep MIDI synths.

ABSTRACT

We present a novel system that gets as an input video frames of a musician playing the piano and generates the music for that video. Generation of music from visual cues is a challenging problem and it is not clear whether it is an attainable goal at all. Our main aim in this work is to explore the plausibility of such a transformation and to identify cues and components able to carry the association of sounds with visual events. To achieve the transformation we built a full pipeline named ` extit{Audeo}' containing three components. We first translate the video frames of the keyboard and the musician hand movements into raw mechanical musical symbolic representation Piano-Roll (Roll) for each video frame which represents the keys pressed at each time step. We then adapt the Roll to be amenable for audio synthesis by including temporal correlations. This step turns out to be critical for meaningful audio generation. As a last step, we implement Midi synthesizers to generate realistic music. extit{Audeo} converts video to audio smoothly and clearly with only a few setup constraints. We evaluate extit{Audeo} on `in the wild' piano performance videos and obtain that their generated music is of reasonable audio quality and can be successfully recognized with high precision by popular music identification software.

Motivation & Objective

• Understand whether visual cues from piano performance can plausibly generate corresponding audio.
• Develop an interpretable, end-to-end pipeline that maps video frames to a musical representation suitable for synthesis.
• Identify visual, symbolic, and synthesis components that enable robust audio generation from unconstrained videos.
• Evaluate the system on real-world piano videos and assess recognizability of generated music using standard tools.

Proposed method

• Video2Roll Net predicts pressed keys per frame by processing five consecutive grayscale frames with a multi-scale feature attention network built on ResNet18, outputting a Piano-Roll M for each middle frame.
• Roll2Midi Net refines the predicted Roll with a GAN (generator G, discriminator D) to produce a robust Pseudo-MIDI representation suitable for synthesis.
• Midi Synth uses classical MIDI synthesis (FluidSynth) and a deep synthesizer (PerfNet-based spectrogram refinement) to convert MIDI into audio via spectrograms and Griffin-Lim, with an optional spectrogram refinement step.

Experimental results

Research questions

• RQ1 Can visual cues from top-view piano videos be translated into a coherent MIDI-like representation that supports audio synthesis?
• RQ2 Does a three-stage pipeline (Video2Roll → Roll2Midi → Midi Synth) produce audible music that aligns with the visual performance and is detectable by music identification software?
• RQ3 What is the impact of using GAN-based refinement on MIDI prediction quality and downstream audio synthesis?
• RQ4 How do traditional and deep synthesizers compare in producing recognizably accurate music from predicted MIDI?
• RQ5 How well does the system generalize to unconstrained videos and unseen composers?

Key findings

• Video2Roll Net achieves higher recall and competitive precision for frame-level key prediction compared with prior work, improving MIDI accuracy.
• Roll2Midi Net improves MIDI predictions by reducing false positives and negatives, boosting F1 score over prior methods.
• Midi synthesis with FluidSynth or PerfNet-based spectrogram refinement yields music that SoundHound detects with higher accuracy for Midi+FluidSynth and Midi+PerfNet configurations than baselines.
• Across evaluations, Midi+FluidSynth achieves the best overall audio detection rate (85.6%) on the main test set, approaching ground-truth performance (87.7%).
• The system demonstrates robustness to performance variations (tempo, legato, staccato) and generalizes to unseen composers, with detection rates around 70% for non-training styles.
• Compared to ResNet baselines, the full Audeo pipeline (Video2Roll + Roll2Midi + Midi Synth) substantially outperforms in musically pertinent metrics.