[Paper Review] DeepSinger: Singing Voice Synthesis with Data Mined From the Web

Yi Ren, Xu Tan+4 more|arXiv (Cornell University)|Jul 9, 2020

Speech Recognition and Synthesis38 references4 citations

TL;DR

DeepSinger is a multilingual, multi-singer singing voice synthesis system that trains entirely on web-mined singing data, bypassing costly human recordings and manual alignments. It uses a novel lyrics-to-singing alignment model and a feed-forward Transformer with a reference encoder to generate high-quality, natural-sounding singing voices directly from lyrics, achieving strong pitch accuracy and voice quality even with noisy training data.

ABSTRACT

In this paper, we develop DeepSinger, a multi-lingual multi-singer singing voice synthesis (SVS) system, which is built from scratch using singing training data mined from music websites. The pipeline of DeepSinger consists of several steps, including data crawling, singing and accompaniment separation, lyrics-to-singing alignment, data filtration, and singing modeling. Specifically, we design a lyrics-to-singing alignment model to automatically extract the duration of each phoneme in lyrics starting from coarse-grained sentence level to fine-grained phoneme level, and further design a multi-lingual multi-singer singing model based on a feed-forward Transformer to directly generate linear-spectrograms from lyrics, and synthesize voices using Griffin-Lim. DeepSinger has several advantages over previous SVS systems: 1) to the best of our knowledge, it is the first SVS system that directly mines training data from music websites, 2) the lyrics-to-singing alignment model further avoids any human efforts for alignment labeling and greatly reduces labeling cost, 3) the singing model based on a feed-forward Transformer is simple and efficient, by removing the complicated acoustic feature modeling in parametric synthesis and leveraging a reference encoder to capture the timbre of a singer from noisy singing data, and 4) it can synthesize singing voices in multiple languages and multiple singers. We evaluate DeepSinger on our mined singing dataset that consists of about 92 hours data from 89 singers on three languages (Chinese, Cantonese and English). The results demonstrate that with the singing data purely mined from the Web, DeepSinger can synthesize high-quality singing voices in terms of both pitch accuracy and voice naturalness (footnote: Our audio samples are shown in https://speechresearch.github.io/deepsinger/.)

Motivation & Objective

• To develop a singing voice synthesis system that bypasses expensive human recordings and manual data labeling.
• To mine and utilize large-scale singing data directly from music websites for training.
• To design an automated alignment system that maps lyrics to singing at the phoneme level without human annotation.
• To build a robust singing model capable of learning from noisy, real-world singing data.
• To enable multilingual and multi-singer synthesis using a single unified model.

Proposed method

• Web crawling is used to collect popular songs from music websites in multiple languages.
• Spleeter is applied to separate singing voices from accompaniment in the crawled audio.
• A hierarchical alignment model is trained to predict phoneme-level durations from sentence-level lyrics and audio.
• A feed-forward Transformer-based singing model generates linear spectrograms directly from lyrics, using a reference encoder to model singer timbre.
• Griffin-Lim is used to reconstruct waveforms from the generated spectrograms.
• The model is trained on a dataset of ~92 hours of singing data from 89 singers across Chinese, Cantonese, and English.

Experimental results

Research questions

• RQ1 Can a high-quality singing voice synthesis system be trained entirely on data mined from the web, without studio recordings or manual annotations?
• RQ2 How effective is an automated, multi-level alignment model in replacing human-labeled alignments for SVS?
• RQ3 Can a reference encoder-based model effectively learn singer-specific timbre from noisy, real-world singing data?
• RQ4 Does multilingual pretraining degrade or improve performance on individual languages?
• RQ5 How does the inclusion of web-mined singing data compare to using only TTS data for auxiliary training?

Key findings

• DeepSinger achieves high voice quality with a Mean Opinion Score (MOS) of 3.78 for synthesized singing, close to the reference score of 4.38, indicating strong naturalness.
• The model with a reference encoder outperforms the version without it, especially on noisy training data, with a 3.27 MOS on noisy references versus 3.42 for normal ones.
• The preference score for DeepSinger over a baseline model is 65.00%, significantly higher than the 19.50% for a model trained only on TTS data, proving the importance of singing-specific data.
• Multilingual training does not degrade performance on any individual language, and the model successfully supports cross-lingual singing synthesis.
• The model trained on clean data performs nearly as well as the one trained on noisy data when tested on clean references, showing robustness to data noise.
• The reference encoder enables the model to generalize across different noise levels and singer styles, even when training data is imperfect.