[論文レビュー] ESPnet: End-to-End Speech Processing Toolkit

Shinji Watanabe, Takaaki Hori+10 more|arXiv (Cornell University)|Mar 30, 2018

Speech Recognition and Synthesis参考文献 36被引用数 74

ひとこと要約

tldr: ESPnet は Kaldi風のデータ処理を備え、Chainer と PyTorch 上に構築されたオープンソースのエンドツーエンドASRツールキットであり、ハイブリッドCTC/アテンションモデル、マルチオブジェクティブ訓練、ジョインデコーディング、言語モデル統合をサポート；WSJ、CSJ、HKUST で競争力のある結果を benchmark する。

ABSTRACT

This paper introduces a new open source platform for end-to-end speech processing named ESPnet. ESPnet mainly focuses on end-to-end automatic speech recognition (ASR), and adopts widely-used dynamic neural network toolkits, Chainer and PyTorch, as a main deep learning engine. ESPnet also follows the Kaldi ASR toolkit style for data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments. This paper explains a major architecture of this software platform, several important functionalities, which differentiate ESPnet from other open source ASR toolkits, and experimental results with major ASR benchmarks.

研究の動機と目的

• Motivate the need for a unified end-to-end ASR platform that simplifies training and evaluation.
• Provide a flexible architecture leveraging CTC/attention hybrids for robust end-to-end ASR.
• Offer Kaldi-style data processing and recipes to ease reproducibility and benchmarking.
• Demonstrate competitive performance on major ASR benchmarks (WSJ, CSJ, HKUST).
• Highlight implementation efficiency and scalability (multi-GPU, PyTorch/Chainer backends).

提案手法

• Adopts a hybrid CTC/attention end-to-end ASR framework to jointly train and decode using a single encoder.
• Uses multiobjective training combining L_ctc and L_att with a tunable alpha parameter (L = alpha L_ctc + (1-alpha) L_att).
• Employs warp CTC for faster CTC computation and supports various attention types (location-aware, dot-product, additive, multi-head).
• Implements joint decoding by combining CTC and attention scores in a one-pass beam search.
• Integrates RNNLMs during decoding via shallow fusion (log p_lm) with a beta scaling parameter.
• Provides Kaldi-style data preprocessing and feature extraction, ensuring compatibility with Kaldi recipes and 80-dim log-MMel features (plus pitch).
• Supports multiple backends (Chainer and PyTorch) and simple, compact Python codebase (~5.4K lines) for model and recognition modules.
• Offers end-to-end ASR recipes for WSJ, Librispeech, TED-LIUM, CSJ, AMI, HKUST, VoxForge, CHiME-4/5 to enable standardized benchmarking.

Figure 1: Software architecture of ESPnet.

実験結果

リサーチクエスチョン

• RQ1 Can end-to-end ASR achieve competitive performance with a unified CTC/attention framework across multiple languages and tasks?
• RQ2 Does multiobjective training and joint CTC/attention decoding improve robustness and convergence in end-to-end ASR?
• RQ3 What are the practical benefits (speed, simplicity, reproducibility) of Kaldi-style data preprocessing in an end-to-end toolkit?
• RQ4 How effectively can end-to-end models leverage external language models during decoding?
• RQ5 To what extent can ESPnet scale to adverse/noisy environments and multilingual settings?

主な発見

ツールキット / セットアップ	指標	dev93	eval92	備考
ESPNET (Chainer)	CER	10.1	7.6	Baseline with VGG2-BLSTM (4 BLSTM layers)
ESPNET (Chainer) + BLSTM	CER	8.5	5.9	Deep encoder (6 BLSTM layers)
ESPNET + char-LSTMLM	CER	8.3	5.2	Incorporates character LM
ESPNET + joint decoding	CER	5.5	3.8	Hybrid CTC/attention joint decoding
ESPNET + label smoothing	CER	5.3	3.6	With label smoothing
ESPNET (Chainer)	WER	12.4	8.9	WSJ results

• On WSJ, deeper encoders and integration of char-based LMs and joint decoding progressively improve CER and WER, with joint decoding achieving CER 5.5 (dev93) / 3.8 (eval92) and WER 12.4 (dev93) / 8.9 (eval92).
• ESPnet with PyTorch backend trains much faster (5 hours on one GPU) than some baselines, and 20 hours on Chainer, highlighting efficiency gains.
• CSJ results show ESPnet achieving CERs of 8.7/6.2/6.9 (eval1/eval2/eval3) with multi-GPU setup providing small improvements (e.g., 8.5/6.1/6.8).
• HKUST Mandarin CTS results show ESPnet approaching state-of-the-art HMM/DNN systems, with CER 28.3 compared to 28.2–34.8 in competing methods.
• Overall, ESPnet delivers competitive end-to-end ASR performance across WSJ, CSJ, and HKUST, sometimes matching or surpassing lattice-free MMI-based or hybrid systems under certain configurations.
• The framework emphasizes simplicity and accessibility, achieving comparable performance with significantly reduced codebase size (~5.4K Python lines) compared to Kaldi and Julius.

Figure 2: Experimental flow of standard ESPnet recipe.