[Paper Review] FunASR: A Fundamental End-to-End Speech Recognition Toolkit

Zhifu Gao, Zerui Li+9 more|arXiv (Cornell University)|May 18, 2023

Speech Recognition and Synthesis4 citations

TL;DR

FunASR is an open-source end-to-end speech recognition toolkit that bridges academic research and industrial deployment by providing high-accuracy, large-scale pre-trained models—especially Paraformer, a non-autoregressive model trained on 60,000 hours of Mandarin speech—alongside optimized inference pipelines, VAD, and punctuation models. It achieves state-of-the-art performance with 1.95% CER on AISHELL and 12x faster inference than autoregressive models, enabling efficient, high-precision deployment on CPU and GPU platforms.

ABSTRACT

This paper introduces FunASR, an open-source speech recognition toolkit designed to bridge the gap between academic research and industrial applications. FunASR offers models trained on large-scale industrial corpora and the ability to deploy them in applications. The toolkit's flagship model, Paraformer, is a non-autoregressive end-to-end speech recognition model that has been trained on a manually annotated Mandarin speech recognition dataset that contains 60,000 hours of speech. To improve the performance of Paraformer, we have added timestamp prediction and hotword customization capabilities to the standard Paraformer backbone. In addition, to facilitate model deployment, we have open-sourced a voice activity detection model based on the Feedforward Sequential Memory Network (FSMN-VAD) and a text post-processing punctuation model based on the controllable time-delay Transformer (CT-Transformer), both of which were trained on industrial corpora. These functional modules provide a solid foundation for building high-precision long audio speech recognition services. Compared to other models trained on open datasets, Paraformer demonstrates superior performance.

Motivation & Objective

• To close the gap between academic research and industrial deployment in end-to-end speech recognition.
• To provide high-precision, large-scale pre-trained models trained on industrial corpora, especially for Mandarin.
• To enable efficient, real-world deployment via optimized inference backends (ONNX, TensorRT, libtorch) and quantization.
• To support domain adaptation through easy fine-tuning on small-scale, domain-specific data.
• To integrate auxiliary modules like VAD and punctuation restoration for full ASR pipeline deployment.

Proposed method

• The core model, Paraformer, is a non-autoregressive end-to-end ASR model trained on 60,000 hours of manually transcribed Mandarin speech.
• Timestamp prediction and hotword customization are added to the Paraformer backbone to enhance recognition accuracy for specific terms and temporal alignment.
• A voice activity detection (VAD) model based on FSMN-VAD is open-sourced, trained on industrial data for robust speech detection.
• A text post-processing model using CT-Transformer is provided for punctuation prediction and disfluency correction.
• The toolkit supports training from scratch and fine-tuning via recipe-based pipelines using PyTorch, with support for AISHELL, LibriSpeech, and WenetSpeech datasets.
• Inference is accelerated via AMP quantization and multiple backends (ONNX, libtorch, TensorRT), enabling CPU and GPU deployment.

Experimental results

Research questions

• RQ1 Can a non-autoregressive end-to-end model trained on large-scale industrial speech data outperform existing models on Mandarin ASR benchmarks?
• RQ2 How effective is fine-tuning a large pre-trained model on small, domain-specific datasets for improving recognition accuracy and keyword recall?
• RQ3 Can auxiliary modules like VAD and punctuation restoration significantly improve the robustness and usability of end-to-end ASR systems in real-world applications?
• RQ4 What is the inference efficiency gain of non-autoregressive models like Paraformer compared to autoregressive models in real deployment scenarios?
• RQ5 To what extent does automatic mixed precision (AMP) quantization improve inference speed without degrading recognition accuracy?

Key findings

• Paraformer-large achieves a 1.95% CER on the AISHELL test set, outperforming Conformer and demonstrating the benefit of large-scale industrial training data.
• The model achieves a 12x speedup in inference on V100 GPU compared to autoregressive models, with RTF dropping from 0.1026 to 0.0446 under INT8 quantization.
• Fine-tuning on 200 hours of domain-specific data (logistics) improved CER from 13.1% to 12.6% and increased domain keyword recall from 74.0% to 96.0% on long test sets.
• The CT-Transformer model achieves competitive F1 scores (70.5) for punctuation and disfluency correction with 1.9x faster inference than full Transformer baselines.
• AMP quantization reduces inference time by 40% (RTF from 0.1026 to 0.0446) without accuracy loss, enabling efficient CPU deployment.
• The toolkit enables high-precision, low-latency deployment across CPU, GPU, and mobile platforms via ONNX, libtorch, and TensorRT backends.