[Paper Review] Short utterance compensation in speaker verification via cosine-based teacher-student learning of speaker embeddings

Jee-weon Jung, Hee-Soo Heo+2 more|arXiv (Cornell University)|Oct 25, 2018

Speech Recognition and Synthesis23 references4 citations

TL;DR

This paper proposes a cosine-based teacher-student learning framework that compensates for short utterance degradation in text-independent speaker verification by leveraging phonetic-level features (130 ms segments) within a raw waveform CNN-GRU network. By jointly optimizing Kullback-Leibler divergence of output layers and cosine distance of speaker embeddings, the method achieves over 65% compensation of performance loss on 2.05s utterances, outperforming conventional post-hoc feature transformation approaches.

ABSTRACT

The short duration of an input utterance is one of the most critical threats that degrade the performance of speaker verification systems. This study aimed to develop an integrated text-independent speaker verification system that inputs utterances with short duration of 2 seconds or less. We propose an approach using a teacher-student learning framework for this goal, applied to short utterance compensation for the first time in our knowledge. The core concept of the proposed system is to conduct the compensation throughout the network that extracts the speaker embedding, mainly in phonetic-level, rather than compensating via a separate system after extracting the speaker embedding. In the proposed architecture, phonetic-level features where each feature represents a segment of 130 ms are extracted using convolutional layers. A layer of gated recurrent units extracts an utterance-level feature using phonetic-level features. The proposed approach also adopts a new objective function for teacher-student learning that considers both Kullback-Leibler divergence of output layers and cosine distance of speaker embeddings layers. Experiments were conducted using deep neural networks that take raw waveforms as input, and output speaker embeddings on VoxCeleb1 dataset. The proposed model could compensate approximately 65 \% of the performance degradation due to the shortened duration.

Motivation & Objective

• To address the critical performance degradation in speaker verification caused by short utterances (≤2s), which lack sufficient phonetic information.
• To overcome limitations of prior methods that compensate utterance-level features after extraction, which cannot recover missing phonetic content.
• To develop an integrated, end-to-end system that compensates for short utterances directly within the speaker embedding extraction network.
• To improve discriminative power and robustness of speaker embeddings by jointly optimizing output layer divergence and speaker embedding similarity.

Proposed method

• Uses a raw waveform CNN-GRU architecture with leaky ReLU and GRU layers to extract speaker embeddings directly from input waveforms.
• Defines phonetic-level features as outputs of the final convolutional layer, representing 130 ms segments of speech, to capture phonetic content.
• Applies a teacher-student learning framework where the teacher network processes long utterances and the student network processes short utterances.
• Introduces a novel objective function combining Kullback-Leibler divergence between output layers and cosine distance between speaker embedding layers.
• Uses cosine similarity between student and teacher speaker embeddings as a key regularization signal to align compact, discriminative embeddings.
• Trains the student network to minimize both the KL divergence of final layer outputs and the cosine distance of speaker embeddings, enhancing robustness and accuracy.

Experimental results

Research questions

• RQ1 Can phonetic-level features extracted from short utterances (130 ms segments) effectively compensate for performance degradation in speaker verification?
• RQ2 Does joint optimization of output layer divergence and speaker embedding similarity improve short utterance robustness compared to conventional teacher-student learning?
• RQ3 Can an end-to-end, integrated system that compensates within the embedding extraction network outperform post-hoc feature transformation methods?
• RQ4 How invariant is the proposed method to varying short utterance durations (e.g., 1.55s to 2.55s)?
• RQ5 Does comparing speaker embeddings directly via cosine distance enhance discriminative power more than relying solely on output layer losses?

Key findings

• The proposed method achieved an equal error rate (EER) of 10.08% on 2.05s utterances, compensating for over 65% of the performance degradation compared to the baseline (12.80% EER).
• Using both KL divergence of output layers and cosine distance of speaker embeddings yielded the best performance, outperforming methods using only one objective.
• The system maintained consistent performance across varying utterance durations, with EER increasing only slightly from 10.08% to 10.40% when the duration range widened to ±0.5s.
• The cosine-based embedding comparison significantly improved performance over MSE-based or KL-divergence-only baselines, indicating its effectiveness in preserving speaker identity structure.
• The method demonstrated that in-network compensation via phonetic-level features is more effective than post-extraction feature transformation, as it preserves discriminative information during embedding learning.
• The ablation study confirmed that joint optimization of output and embedding space alignment is essential, as using only one loss term led to suboptimal results.