[Paper Review] AISHELL-3: A Multi-speaker Mandarin TTS Corpus and the Baselines

Yao Shi, Hui Bu+3 more|arXiv (Cornell University)|Oct 22, 2020

Speech Recognition and SynthesisComputer Science23 references82 citations

TL;DR

AISHELL-3 provides a large-scale Mandarin multi-speaker speech corpus (≈85 hours, 218 speakers) with Chinese characters and pinyin transcripts, plus a baseline multi-speaker TTS system with speaker-embedding feedback to enable zero-shot voice cloning.

ABSTRACT

In this paper, we present AISHELL-3, a large-scale and high-fidelity multi-speaker Mandarin speech corpus which could be used to train multi-speaker Text-to-Speech (TTS) systems. The corpus contains roughly 85 hours of emotion-neutral recordings spoken by 218 native Chinese mandarin speakers. Their auxiliary attributes such as gender, age group and native accents are explicitly marked and provided in the corpus. Accordingly, transcripts in Chinese character-level and pinyin-level are provided along with the recordings. We present a baseline system that uses AISHELL-3 for multi-speaker Madarin speech synthesis. The multi-speaker speech synthesis system is an extension on Tacotron-2 where a speaker verification model and a corresponding loss regarding voice similarity are incorporated as the feedback constraint. We aim to use the presented corpus to build a robust synthesis model that is able to achieve zero-shot voice cloning. The system trained on this dataset also generalizes well on speakers that are never seen in the training process. Objective evaluation results from our experiments show that the proposed multi-speaker synthesis system achieves high voice similarity concerning both speaker embedding similarity and equal error rate measurement. The dataset, baseline system code and generated samples are available online.

Motivation & Objective

• Provide a large, open Mandarin multi-speaker speech corpus for TTS research.
• Enable training of multi-speaker TTS systems with explicit speaker attributes (gender, age, accent).
• Demonstrate a baseline multi-speaker TTS system using speaker embeddings and feedback constraints.
• Investigate data preparation and augmentation strategies to improve model robustness and generalization.
• Evaluate speaker similarity and generalization to unseen speakers using objective metrics.

Proposed method

• Introduce AISHELL-3 dataset with 85 hours, 218 native Mandarin speakers, 88,035 recordings, and transcripts in Chinese characters and pinyin.
• Develop a speaker-agnostic textual frontend and a speaker-aware acoustic model based on Tacotron-2 with a speaker encoder for voice conditioning.
• Incorporate a speaker-embedding feedback constraint by adding a cosine similarity loss between reference and synthesized speaker embeddings.
• Use a ResNet-based speaker encoder with global mean-std pooling to obtain fixed-dimensional speaker embeddings.
• Apply data preparation techniques including prosodic label prediction, silence trimming, and long-form sentence augmentation to improve alignment and generalization.
• Train and evaluate objective measures of speaker similarity (cosine similarity, SV-EER) on seen and unseen speakers.

Experimental results

Research questions

• RQ1 Can AISHELL-3 support effective training of multi-speaker Mandarin TTS systems including zero-shot voice cloning?
• RQ2 How does a speaker-embedding feedback constraint affect speaker similarity and robustness for unseen speakers?
• RQ3 What data preparation and augmentation strategies improve alignment, prosody, and long-form synthesis in Mandarin TTS?
• RQ4 How well does the baseline system generalize from seen to unseen speakers compared with English multi-speaker corpora?
• RQ5 What objective measures reflect speaker similarity and voice identity in synthesized Mandarin speech?

Key findings

• AISHELL-3 contains 85 hours of Mandarin speech from 218 speakers with gender, age, and accent annotations and transcripts in Chinese characters and pinyin.
• The baseline multi-speaker TTS system (Tacotron-2 with a speaker encoder and embedding feedback) achieves high speaker similarity as shown by cosine similarity and SV-EER metrics on seen and unseen speakers.
• Obj. evaluations show the model maintains speaker similarity when generalizing to unseen speakers, with EER changes consistent with prior English VCTK-based work.
• Data augmentation and preprocessing (prosody labeling, silence trimming, long-form sentence concatenation) improve training efficiency and alignment during Tacotron-2 optimization.
• The model uses a frozen speaker encoder and a cosine-similarity loss term to reinforce voice similarity during training (alpha = 1.0).
• Results align with prior VCTK-based studies, indicating the AISHELL-3 baseline supports robust multi-speaker Mandarin synthesis and zero-shot voice cloning.