[Paper Review] Finding phonemes: improving machine lip-reading

Helen L. Bear, Richard Harvey+1 more|arXiv (Cornell University)|Oct 3, 2017

Speech and Audio Processing7 references5 citations

TL;DR

This paper proposes a speaker-dependent, data-driven method to derive optimal viseme units for machine lip-reading by clustering phonemes based on confusion matrices from phoneme recognition. Using the LiLIR dataset and HMM-based classification, it demonstrates that word recognition performance is often better with phoneme-level classifiers than with traditional viseme sets, and that intermediate units between phonemes and visemes yield the best results, challenging the conventional wisdom that visemes are optimal.

ABSTRACT

In machine lip-reading there is continued debate and research around the correct classes to be used for recognition. In this paper we use a structured approach for devising speaker-dependent viseme classes, which enables the creation of a set of phoneme-to-viseme maps where each has a different quantity of visemes ranging from two to 45. Viseme classes are based upon the mapping of articulated phonemes, which have been confused during phoneme recognition, into viseme groups. Using these maps, with the LiLIR dataset, we show the effect of changing the viseme map size in speaker-dependent machine lip-reading, measured by word recognition correctness and so demonstrate that word recognition with phoneme classifiers is not just possible, but often better than word recognition with viseme classifiers. Furthermore, there are intermediate units between visemes and phonemes which are better still.

Motivation & Objective

• To investigate whether viseme-based classification is optimal for visual-only speech recognition or if intermediate units between phonemes and visemes yield better performance.
• To develop a systematic, speaker-dependent method for generating custom viseme maps from phoneme confusion data, enabling controlled comparison across different viseme set sizes.
• To evaluate word recognition accuracy across varying numbers of visual units (from 2 to 45 visemes) to identify the optimal number and structure of viseme classes per speaker.
• To challenge the conventional assumption that visemes are the best intermediate representation in lip-reading by quantitatively comparing performance against phoneme-level and viseme-level classifiers.

Proposed method

• The method begins with speaker-specific phoneme recognition using 10-fold cross-validation and HTK-based HMMs, producing confusion matrices from 200 utterances per speaker.
• Confusion matrices are column-normalized to obtain conditional probabilities Pr{p_i | p̂_j}, representing the likelihood that a true phoneme p_i was misclassified as p̂_j.
• A greedy clustering algorithm merges the two most confused phonemes (highest q = Pr{p_r|p̂_s} + Pr{p_s|p̂_r}) into new viseme classes, repeating until only two classes remain.
• Vowels and consonants are clustered separately to preserve articulatory distinctions, ensuring viseme groups are phonetically coherent.
• Each intermediate clustering step (from 45 down to 2 visemes) generates a new viseme map, which is then used to retrain HMM-based word recognition models.
• Performance is measured by word recognition correctness, normalized across training data variance, to compare the impact of different viseme set sizes.

Experimental results

Research questions

• RQ1 Does using phoneme-level classification in lip-reading outperform traditional viseme-based classification?
• RQ2 Is there an optimal number of visual units between phonemes and visemes that maximizes word recognition accuracy?
• RQ3 Do conventional viseme sets (e.g., Lee set, Jeffers set) represent the best configuration for individual speakers?
• RQ4 How do the number of homophones and insufficient training data per unit affect recognition performance as viseme set size varies?
• RQ5 Can speaker-specific viseme maps derived from confusion matrices outperform fixed, cross-speaker viseme definitions?

Key findings

• Word recognition performance with phoneme-level classifiers significantly outperforms that of viseme-level classifiers for all 12 speakers in the LiLIR dataset.
• The optimal number of visual units per speaker lies between 8 and the number of phonemes, with no consistent match to conventional viseme sets (e.g., 10–20 visemes).
• Performance curves are non-monotonic: correctness drops sharply below 10 visemes due to increased homophones, and also drops at high viseme counts due to data sparsity and low visual distinctiveness.
• For three out of twelve speakers, the best-performing viseme set size fell within the 10–20 range, but this was not statistically better than chance, indicating no universal optimal size.
• The best-performing viseme maps were speaker-specific, with no consistent pattern across speakers, suggesting that cross-speaker viseme sets are unlikely to be optimal.
• The study identifies specific viseme class merges (e.g., /s/ and /r/ in SP01) that significantly improve recognition, confirming that optimal viseme grouping is highly individualized.