[Paper Review] The CAPIO 2017 Conversational Speech Recognition System

Kyu J. Han, Akshay Chandrashekaran+2 more|arXiv (Cornell University)|Dec 29, 2017

Speech Recognition and Synthesis5 references77 citations

TL;DR

The CAPIO 2017 system introduces densely connected LSTMs (dense LSTMs) and a simple acoustic model adaptation by parameter averaging, achieving state-of-the-art WER on NIST 2000 Hub5 English. It also demonstrates strong results on non-telephony data with TED-LIUM and LibriSpeech.

ABSTRACT

In this paper we show how we have achieved the state-of-the-art performance on the industry-standard NIST 2000 Hub5 English evaluation set. We explore densely connected LSTMs, inspired by the densely connected convolutional networks recently introduced for image classification tasks. We also propose an acoustic model adaptation scheme that simply averages the parameters of a seed neural network acoustic model and its adapted version. This method was applied with the CallHome training corpus and improved individual system performances by on average 6.1% (relative) against the CallHome portion of the evaluation set with no performance loss on the Switchboard portion. With RNN-LM rescoring and lattice combination on the 5 systems trained across three different phone sets, our 2017 speech recognition system has obtained 5.0% and 9.1% on Switchboard and CallHome, respectively, both of which are the best word error rates reported thus far. According to IBM in their latest work to compare human and machine transcriptions, our reported Switchboard word error rate can be considered to surpass the human parity (5.1%) of transcribing conversational telephone speech.

Motivation & Objective

• Develop densely connected LSTM architectures to mitigate vanishing gradients in deep acoustic models for conversational speech recognition.
• Propose a scalable acoustic model adaptation scheme based on averaging seed and adapted model parameters.
• Explore diverse acoustic models (dense LSTM and CNN-bLSTM) across multiple phonesets and hesitation modeling to improve system robustness.
• Demonstrate system performance through extensive experimentation on telephony (Switchboard/CallHome) and non-telephony (TED-LIUM/LibriSpeech) datasets.
• Show that dense architectures plus system combination yield state-of-the-art WER on standard benchmarks.

Proposed method

• Propose densely connected LSTM architectures (dense LSTMs) that concatenate outputs from all previous layers within a dense block to combat gradient vanishing.
• Implement two dense LSTM variants: dense TDNN-LSTM and dense CNN-bLSTM, evaluated with LF-MMI training and SAT/fMLLR preprocessing.
• Apply acoustic model adaptation by averaging parameters of a seed model and its adapted version (parameter averaging across GPUs during adaptation).
• Use multiple phonesets (PronLex, CMU, MSU) and hesitation modeling (2 vs 11 hesitations) to build diverse CNN-bLSTMs; train and compare across configurations.
• Train language models (4-gram and RNN LMs) and perform lattice-based minimum Bayes risk system combination across diverse systems.

Experimental results

Research questions

• RQ1 Can densely connected LSTMs enable deeper LSTM networks for speech recognition without gradient vanishing, improving WER over residual connections?
• RQ2 Does a simple parameter-averaging adaptation reliably improve domain-specific (CallHome) performance without degrading Switchboard performance?
• RQ3 How do different dense LSTM and CNN-bLSTM configurations perform across multiple phonesets and hesitation models on Switchboard and CallHome?
• RQ4 What is the impact of system combination of diverse models on achieving state-of-the-art WER on Hub5 English benchmarks?
• RQ5 Do the proposed dense architectures generalize to non-telephony datasets like TED-LIUM and LibriSpeech?

Key findings

• Dense LSTMs continuously improve as more layers are added, achieving lower WER than residual LSTMs beyond 10 layers.
• Dense CNN-bLSTM and dense TDNN-LSTM variants show substantial WER reductions over non-dense baselines, with PronLex/2 hesitations yielding strong results.
• Acoustic model adaptation with parameter averaging improves CallHome WER by about 5% relative on average while preserving Switchboard performance.
• System combination of eight diverse systems yields 5.0% Switchboard and 9.1% CallHome WER, the best reported for this setup at the time.
• RNN LM rescoring provides consistent improvements across configurations, with up to 8% relative reduction in Switchboard WER for certain dense models.
• For non-telephony data (TED-LIUM LibriSpeech), the dense architectures contribute to state-of-the-art results for the respective corpora when combined.