[Paper Review] Weakly Supervised Multi-Embeddings Learning of Acoustic Models

Gabriel Synnaeve, Emmanuel Dupoux|arXiv (Cornell University)|Dec 20, 2014

Speech Recognition and Synthesis11 references6 citations

TL;DR

This paper proposes a weakly supervised Siamese neural network that jointly learns phone and speaker embeddings using only same/different word and speaker labels. By training on a subset of the Buckeye corpus with multi-task loss, the model achieves strong performance in both phoneme and speaker discrimination, demonstrating that shared representations can be effectively learned with minimal supervision, and revealing that speaker identity is encoded more sparsely and specifically than phoneme identity in the embedding space.

ABSTRACT

We trained a Siamese network with multi-task same/different information on a speech dataset, and found that it was possible to share a network for both tasks without a loss in performance. The first task was to discriminate between two same or different words, and the second was to discriminate between two same or different talkers.

Motivation & Objective

• To investigate whether joint learning of phone and speaker embeddings is feasible using only weak supervision (same/different labels).
• To evaluate if a shared Siamese network architecture can maintain performance on both phoneme and speaker discrimination tasks without performance degradation.
• To analyze the nature of information encoding in hidden and embedding layers, particularly the degree of specialization for phoneme vs. speaker identity.
• To assess whether such embeddings could serve as a viable alternative or complement to i-vectors in downstream speech recognition tasks.

Proposed method

• A Siamese network architecture is used, with two parallel streams processing 11-frame log-mel filterbank features.
• The network has three hidden layers of 500 units with sigmoid activations, producing two 100-dimensional embeddings per input: one for word identity (phone), one for speaker identity.
• A multi-task loss function combines cosine-similarity-based losses for both word and speaker same/different classification, using a sum of two loss terms.
• The loss function uses a margin-based approach: for same pairs, it minimizes 1 - cos(similarity); for different pairs, it maximizes cos²(similarity).
• Training uses Adadelta optimization with adaptive learning rates, early stopping on a 10% held-out development set.
• Information encoding is analyzed via F-test on between- and within-class variance ratios across units in each layer, classifying units as phone-specific, speaker-specific, or dual-specific.

Experimental results

Research questions

• RQ1 Can a single Siamese network effectively learn both phone and speaker embeddings using only same/different word and speaker labels?
• RQ2 How does joint training affect the performance of each individual task compared to single-task training?
• RQ3 What is the degree of specialization in hidden and embedding layers for phoneme versus speaker identity?
• RQ4 How does the sparsity and distribution of coding units evolve across network layers?
• RQ5 Can the learned embeddings support downstream tasks like phoneme discrimination or speaker verification?

Key findings

• The multi-task model achieved strong performance on both phoneme and speaker discrimination tasks, with no degradation compared to single-task baselines.
• Speaker discrimination performance was higher in the multi-task setup than in the filterbank baseline, indicating that phonetic information aids speaker recognition.
• The phone embedding layer showed lower performance (64.8% ABX accuracy) compared to the speaker embedding (85.1% ABX accuracy), suggesting that phoneme coding is less robust in the current setup.
• Hidden layer analysis revealed that early layers predominantly encode phoneme information, while later layers show increasing speaker-specific coding, with a decline in dual-coding units.
• The speaker embedding layer exhibited highly sparse and specific coding, whereas the phone embedding layer contained more doubly-used and less specific units, indicating interference from speaker variability.
• Filterbank coefficients showed that lower-frequency bands were more sensitive to phonetic content, while higher-frequency bands were more sensitive to speaker characteristics.