[Paper Review] ESPnet: End-to-End Speech Processing Toolkit

Shinji Watanabe, Takaaki Hori+10 more|arXiv (Cornell University)|Mar 30, 2018

Speech Recognition and SynthesisComputer Science36 references74 citations

TL;DR

ESPnet presents an open-source end-to-end ASR toolkit built on Chainer and PyTorch with Kaldi-like data processing, supporting hybrid CTC/attention models, multiobjective training, joint decoding, and language model integration; it benchmarks on WSJ, CSJ, and HKUST showing competitive results.

ABSTRACT

This paper introduces a new open source platform for end-to-end speech processing named ESPnet. ESPnet mainly focuses on end-to-end automatic speech recognition (ASR), and adopts widely-used dynamic neural network toolkits, Chainer and PyTorch, as a main deep learning engine. ESPnet also follows the Kaldi ASR toolkit style for data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments. This paper explains a major architecture of this software platform, several important functionalities, which differentiate ESPnet from other open source ASR toolkits, and experimental results with major ASR benchmarks.

Motivation & Objective

• Motivate the need for a unified end-to-end ASR platform that simplifies training and evaluation.
• Provide a flexible architecture leveraging CTC/attention hybrids for robust end-to-end ASR.
• Offer Kaldi-style data processing and recipes to ease reproducibility and benchmarking.
• Demonstrate competitive performance on major ASR benchmarks (WSJ, CSJ, HKUST).
• Highlight implementation efficiency and scalability (multi-GPU, PyTorch/Chainer backends).

Proposed method

• Adopts a hybrid CTC/attention end-to-end ASR framework to jointly train and decode using a single encoder.
• Uses multiobjective training combining L_ctc and L_att with a tunable alpha parameter (L = alpha L_ctc + (1-alpha) L_att).
• Employs warp CTC for faster CTC computation and supports various attention types (location-aware, dot-product, additive, multi-head).
• Implements joint decoding by combining CTC and attention scores in a one-pass beam search.
• Integrates RNNLMs during decoding via shallow fusion (log p_lm) with a beta scaling parameter.
• Provides Kaldi-style data preprocessing and feature extraction, ensuring compatibility with Kaldi recipes and 80-dim log-MMel features (plus pitch).
• Supports multiple backends (Chainer and PyTorch) and simple, compact Python codebase (~5.4K lines) for model and recognition modules.
• Offers end-to-end ASR recipes for WSJ, Librispeech, TED-LIUM, CSJ, AMI, HKUST, VoxForge, CHiME-4/5 to enable standardized benchmarking.

Figure 1: Software architecture of ESPnet.

Experimental results

Research questions

• RQ1 Can end-to-end ASR achieve competitive performance with a unified CTC/attention framework across multiple languages and tasks?
• RQ2 Does multiobjective training and joint CTC/attention decoding improve robustness and convergence in end-to-end ASR?
• RQ3 What are the practical benefits (speed, simplicity, reproducibility) of Kaldi-style data preprocessing in an end-to-end toolkit?
• RQ4 How effectively can end-to-end models leverage external language models during decoding?
• RQ5 To what extent can ESPnet scale to adverse/noisy environments and multilingual settings?

Key findings

• On WSJ, deeper encoders and integration of char-based LMs and joint decoding progressively improve CER and WER, with joint decoding achieving CER 5.5 (dev93) / 3.8 (eval92) and WER 12.4 (dev93) / 8.9 (eval92).
• ESPnet with PyTorch backend trains much faster (5 hours on one GPU) than some baselines, and 20 hours on Chainer, highlighting efficiency gains.
• CSJ results show ESPnet achieving CERs of 8.7/6.2/6.9 (eval1/eval2/eval3) with multi-GPU setup providing small improvements (e.g., 8.5/6.1/6.8).
• HKUST Mandarin CTS results show ESPnet approaching state-of-the-art HMM/DNN systems, with CER 28.3 compared to 28.2–34.8 in competing methods.
• Overall, ESPnet delivers competitive end-to-end ASR performance across WSJ, CSJ, and HKUST, sometimes matching or surpassing lattice-free MMI-based or hybrid systems under certain configurations.
• The framework emphasizes simplicity and accessibility, achieving comparable performance with significantly reduced codebase size (~5.4K Python lines) compared to Kaldi and Julius.

Figure 2: Experimental flow of standard ESPnet recipe.