[Paper Review] An Investigation of the Relation Between Grapheme Embeddings and Pronunciation for Tacotron-based Systems

Antoine Perquin, Erica Cooper+1 more|arXiv (Cornell University)|Oct 21, 2020

Natural Language Processing Techniques16 references4 citations

TL;DR

This paper investigates how Tacotron models trained on French graphemes learn phoneme-like representations in their embeddings, demonstrating that these grapheme embeddings implicitly encode pronunciation information despite no phonetic supervision during training. The key finding is that these embeddings can be used for grapheme-to-phoneme conversion and phonetic control in synthesized speech, enabling phonemic manipulation even in grapheme-only systems.

ABSTRACT

End-to-end models, particularly Tacotron-based ones, are currently a popular solution for text-to-speech synthesis. They allow the production of high-quality synthesized speech with little to no text preprocessing. Indeed, they can be trained using either graphemes or phonemes as input directly. However, in the case of grapheme inputs, little is known concerning the relation between the underlying representations learned by the model and word pronunciations. This work investigates this relation in the case of a Tacotron model trained on French graphemes. Our analysis shows that grapheme embeddings are related to phoneme information despite no such information being present during training. Thanks to this property, we show that grapheme embeddings learned by Tacotron models can be useful for tasks such as grapheme-to-phoneme conversion and control of the pronunciation in synthetic speech.

Motivation & Objective

• To investigate whether Tacotron models trained on graphemes learn representations that correlate with phonemes, despite no explicit phonetic supervision.
• To evaluate the utility of grapheme embeddings for grapheme-to-phoneme (G2P) conversion in low-resource settings.
• To explore the feasibility of phonetic control in grapheme-based TTS systems through embedding manipulation.
• To compare performance of grapheme-based and phoneme-based Tacotron models on a high-quality French dataset.
• To assess the impact of orthographic consistency on the learned embeddings' phonetic relevance.

Proposed method

• A Tacotron-based model was trained on a curated French dataset using one-hot encoded graphemes as input, with a CBHG module and self-attention blocks to enhance sequence modeling.
• Embeddings were extracted from the CBHG module output, which processes variable-length grapheme sequences and acts as a filter-bank for N-gram-like patterns.
• A shallow LSTM model with CTC loss was trained to predict phonemes from the extracted grapheme embeddings, enabling end-to-end phoneme prediction without aligned data.
• Phoneme error rate (PER) was measured on test sets to evaluate G2P performance using embeddings versus raw graphemes.
• A proof-of-concept experiment swapped contextual grapheme embeddings between sentences to control pronunciation in synthesized speech.
• The WaveRNN vocoder was used to generate speech from mel-spectrograms predicted by the Tacotron model.

Experimental results

Research questions

• RQ1 Do grapheme embeddings in a Tacotron model trained without phonetic supervision contain meaningful phonetic information?
• RQ2 Can grapheme embeddings be used effectively for grapheme-to-phoneme conversion, especially in low-resource scenarios?
• RQ3 Is it possible to perform phonetic control in a grapheme-based TTS system by manipulating learned embeddings?
• RQ4 How does the performance of grapheme-based Tacotron compare to phoneme-based Tacotron on a high-quality French dataset?
• RQ5 How does orthographic consistency affect the phonetic relevance of learned grapheme embeddings?

Key findings

• The grapheme-based Tacotron model achieved performance on par with the phoneme-based model in terms of pronunciation and overall speech quality on a well-curated French dataset.
• The Tacotron model's grapheme embeddings achieved a mean phoneme error rate (PER) of 12.8% on the training set when used for G2P conversion, outperforming models trained directly on raw graphemes (19.3% PER).
• On a smaller dev set (400 sentences), the G2P model using embeddings still outperformed the raw grapheme model (16.4% vs. 30.2% PER), suggesting embeddings are more robust to data scarcity.
• Phonetic control was successfully demonstrated by swapping embeddings of 'b' and 'm' graphemes in similar contexts, resulting in synthesized speech that correctly produced /m/ instead of /b/.
• When context differences were large, listeners could not reliably identify the intended phoneme, indicating context dependency of embedding-based control.
• The results suggest that Tacotron’s CBHG module implicitly learns statistical patterns akin to N-gram-based G2P rules, enabling phonetic representation learning from graphemes alone.