[Paper Review] Streaming Small-Footprint Keyword Spotting using Sequence-to-Sequence Models

Yanzhang He, Rohit Prabhavalkar+4 more|arXiv (Cornell University)|Oct 26, 2017

Speech Recognition and Synthesis35 references4 citations

TL;DR

This paper proposes a streaming, small-footprint keyword spotting system using a recurrent neural network transducer (RNN-T) with a novel attention-based keyword biasing technique to improve detection of arbitrary keywords. The RNN-T model jointly learns acoustic and language modeling, and with phoneme targets and an end-of-word token, it achieves a 39% reduction in false reject rate (8.9% vs. 14.5%) at 0.05 false alarms per hour compared to a strong CTC-based baseline.

ABSTRACT

We develop streaming keyword spotting systems using a recurrent neural network transducer (RNN-T) model: an all-neural, end-to-end trained, sequence-to-sequence model which jointly learns acoustic and language model components. Our models are trained to predict either phonemes or graphemes as subword units, thus allowing us to detect arbitrary keyword phrases, without any out-of-vocabulary words. In order to adapt the models to the requirements of keyword spotting, we propose a novel technique which biases the RNN-T system towards a specific keyword of interest. Our systems are compared against a strong sequence-trained, connectionist temporal classification (CTC) based "keyword-filler" baseline, which is augmented with a separate phoneme language model. Overall, our RNN-T system with the proposed biasing technique significantly improves performance over the baseline system.

Motivation & Objective

• To develop a streaming, low-resource keyword spotting system suitable for mobile deployment.
• To enable detection of arbitrary keywords without out-of-vocabulary words by using subword units like phonemes or graphemes.
• To improve detection accuracy for specific keywords using a novel attention-based biasing mechanism in an RNN-T framework.
• To evaluate the performance of RNN-T models trained on phoneme vs. grapheme targets in a streaming KWS setting.
• To compare the proposed RNN-T system against a strong CTC-based keyword-filler baseline with an external language model.

Proposed method

• The system uses a recurrent neural network transducer (RNN-T) to perform end-to-end sequence-to-sequence learning, jointly modeling acoustic and language components.
• The model is trained to predict either phonemes or graphemes as subword units, enabling detection of arbitrary keywords without OOV issues.
• A novel keyword biasing technique uses an attention mechanism that focuses the model’s prediction on a specific keyword during inference.
• The biasing mechanism introduces a learnable <eokw> token and scales attention weights toward the keyword’s phoneme sequence during decoding.
• An end-of-word (<eow>) token is used to signal the end of keyword prediction, improving performance over models without it.
• The system is evaluated using a standard KWS test set with metrics including false reject rate (FR) and false alarm rate (FA) per hour.

Experimental results

Research questions

• RQ1 Can an RNN-T-based sequence-to-sequence model achieve competitive keyword spotting performance in a streaming, low-footprint setting?
• RQ2 How does training on phoneme vs. grapheme targets affect keyword spotting accuracy in RNN-T models?
• RQ3 Does the proposed attention-based keyword biasing mechanism significantly reduce false reject rates compared to a strong CTC-based baseline?
• RQ4 What impact does the <eow> token have on RNN-T performance in keyword spotting?
• RQ5 How does the model’s attention behavior differ between positive and negative keyword utterances?

Key findings

• The RNN-T model trained on phoneme targets with the <eow> token achieves a false reject rate of 11.1% at 0.05 false alarms per hour, outperforming the CTC-based baseline.
• The proposed keyword biasing technique reduces the false reject rate to 8.9% at 0.05 FAs per hour, representing a 39% relative improvement over the baseline.
• The RNN-T model with grapheme targets performs worse than the phoneme-based model, with a false reject rate of 15.5% before and 14.0% after orthographic normalization.
• The attention mechanism in the biased RNN-T system clearly focuses on the keyword phonemes during positive utterances, showing a diagonal pattern in attention weights.
• For negative utterances, attention is dispersed across the keyword’s initial phonemes and the <n/a> token, indicating failure to commit to the keyword.
• The histogram of false reject rates across keywords shows most are below 15%, with only a few outliers, indicating robustness across diverse keywords.