[Paper Review] DALI: a large Dataset of synchronized Audio, LyrIcs and notes, automatically created using teacher-student machine learning paradigm

Gabriel Meseguer-Brocal, Alice Cohen-Hadria+1 more|arXiv (Cornell University)|Jun 25, 2019

Music and Audio Processing26 references47 citations

TL;DR

DALI introduces a large multimodal dataset of 5358 songs with time-aligned vocals and lyrics, created automatically via an iterative teacher-student learning framework to improve Singing Voice Detection and audio-lyrics alignment.

ABSTRACT

The goal of this paper is twofold. First, we introduce DALI, a large and rich multimodal dataset containing 5358 audio tracks with their time-aligned vocal melody notes and lyrics at four levels of granularity. The second goal is to explain our methodology where dataset creation and learning models interact using a teacher-student machine learning paradigm that benefits each other. We start with a set of manual annotations of draft time-aligned lyrics and notes made by non-expert users of Karaoke games. This set comes without audio. Therefore, we need to find the corresponding audio and adapt the annotations to it. To that end, we retrieve audio candidates from the Web. Each candidate is then turned into a singing-voice probability over time using a teacher, a deep convolutional neural network singing-voice detection system (SVD), trained on cleaned data. Comparing the time-aligned lyrics and the singing-voice probability, we detect matches and update the time-alignment lyrics accordingly. From this, we obtain new audio sets. They are then used to train new SVD students used to perform again the above comparison. The process could be repeated iteratively. We show that this allows to progressively improve the performances of our SVD and get better audio-matching and alignment.

Motivation & Objective

• Provide a large, publicly available multimodal dataset of synchronized audio, lyrics, and vocal melody notes with four levels of lyric granularity.
• Describe an iterative teacher-student learning framework that improves singing voice detection (SVD) and audio-annotation alignment.
• Demonstrate that leveraging imperfect but larger training data via teacher-student learning yields better cross-dataset generalization.
• Show how automatic candidate audio retrieval and alignment can scale dataset creation for MIR research.

Proposed method

• Collect manual Karaoke-based annotations (time, notes, text) lacking exact audio version.]
• Retrieve candidate audio tracks from WASABI-linked song information and YouTube; select the best match by aligning a singing-voice probability sequence with annotation voice sequence using NCC.
• Compute singing-voice probability with a ConvNet-based SVD system trained on labeled data; align with annotations by optimizing offset o and frame rate fr via NCC and brute-force fr search.
• Iteratively train a 'student' SVD on the larger set of matches to improve p̂(t) and re-expand the dataset, forming a teacher-student loop to enhance alignment quality.

Experimental results

Research questions

• RQ1 Can a large, automatically constructed multimodal dataset of synchronized audio, lyrics, and notes be created from karaoke annotations and web audio candidates?
• RQ2 Does a teacher-student learning paradigm improve singing voice detection and alignment quality over a single-pass system?
• RQ3 How does cross-dataset generalization behave for SVD models trained on imperfect, large-scale data vs. small, high-quality labeled data?
• RQ4 What is the impact of improved SVD on the size and quality of the DALI dataset?

Key findings

• 5358 songs are included in DALI with full audio and time-aligned lyrics and vocal notes at four granularity levels.
• A ConvNet-based singing voice detector (teacher) selects audio candidates by maximizing cross-correlation with annotation voice sequences (avs).
• Teacher-student experiments show students generally outperform teachers in cross-dataset tests (e.g., Jamendo and MedleyDB).
• Cross-dataset evaluation reveals students trained on one teacher’s output can achieve higher frame accuracy on the other dataset (e.g., 86.5% on both J_test and M_test).
• The dataset creation approach benefits from using larger, imperfect data over smaller, perfect data for deep learning in this context.