[Paper Review] Learning acoustic word embeddings with phonetically associated triplet network

Hyungjun Lim, Younggwan Kim+3 more|arXiv (Cornell University)|Nov 7, 2018

Speech Recognition and Synthesis25 references4 citations

TL;DR

This paper proposes the Phonetically Associated Triplet Network (PATN), a hierarchical multitask learning framework that enhances acoustic word embeddings by jointly optimizing triplet loss and frame-level cross-entropy loss in a bidirectional LSTM architecture. The method achieves over 20% relative improvement in recall at 1.0 false alarm per hour on wake-up word detection, with strong generalization to out-of-domain data.

ABSTRACT

Previous researches on acoustic word embeddings used in query-by-example spoken term detection have shown remarkable performance improvements when using a triplet network. However, the triplet network is trained using only a limited information about acoustic similarity between words. In this paper, we propose a novel architecture, phonetically associated triplet network (PATN), which aims at increasing discriminative power of acoustic word embeddings by utilizing phonetic information as well as word identity. The proposed model is learned to minimize a combined loss function that was made by introducing a cross entropy loss to the lower layer of LSTM-based triplet network. We observed that the proposed method performs significantly better than the baseline triplet network on a word discrimination task with the WSJ dataset resulting in over 20% relative improvement in recall rate at 1.0 false alarm per hour. Finally, we examined the generalization ability by conducting the out-of-domain test on the RM dataset.

Motivation & Objective

• To improve discriminative power of acoustic word embeddings for query-by-example spoken term detection.
• To address the limitation of triplet networks that ignore phonetic information and rely only on word-level similarity.
• To enhance model generalization to out-of-domain data, especially for wake-up word detection.
• To integrate phonetic-level supervision into the triplet network via hierarchical multitask learning.

Proposed method

• Proposes a phonetically associated triplet network (PATN) that combines triplet loss with frame-level cross-entropy loss in a shared bidirectional LSTM architecture.
• Introduces a joint loss function: $\mathcal{L}_{PT} = (1-\lambda)\mathcal{L}_T + \lambda\mathcal{L}_{CE}$, where $\lambda$ controls the trade-off between word-level and frame-level learning.
• Applies cross-entropy loss at the lower LSTM layer to model phonetic variations, while triplet loss at the higher layer enforces inter-class and intra-class embedding separation.
• Uses concatenation of bidirectional LSTM hidden states as the final acoustic word embedding for cosine similarity computation.
• Employs t-SNE visualization to qualitatively assess embedding clustering and separability.
• Trains on WSJ dataset and evaluates on both in-domain (WSJ) and out-of-domain (RM) test sets.

Experimental results

Research questions

• RQ1 Can joint learning of phonetic and word-level criteria improve the discriminative quality of acoustic word embeddings?
• RQ2 Does incorporating frame-level phonetic supervision enhance generalization to out-of-domain wake-up word detection?
• RQ3 How does the proposed PATN model compare to standard triplet networks in terms of recall and false alarm rate?
• RQ4 Can the model maintain performance improvements without increasing model size?

Key findings

• The proposed PATN achieved a recall of 0.714 at 1.0 false alarm per hour on the in-domain WSJ dataset, representing over 20% relative improvement over the baseline triplet network.
• On the out-of-domain RM dataset, PATN achieved a recall of 0.582, a significant improvement over the baseline’s 0.463, demonstrating strong generalization.
• t-SNE visualization showed that PATN produces more separable and less confusable word clusters, especially for phonetically similar words like 'give' vs. 'get' and 'have' vs. 'how'.
• The model with two BLSTM layers and $\lambda=0.1$ achieved the best performance, indicating optimal trade-off between triplet and cross-entropy losses.
• The improvement was achieved without increasing model size, confirming the effectiveness of the joint learning strategy.