[Paper Review] RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognition

Peiyan Dong, Siyue Wang+9 more|arXiv (Cornell University)|Feb 19, 2020

Advanced Neural Network Applications35 references12 citations

TL;DR

RTMobile is the first framework to achieve real-time RNN inference on mobile devices by combining block-based structured pruning (BSP) with compiler optimizations. It enables over 10× model compression with no accuracy loss and achieves 40× higher energy efficiency than prior FPGA-based methods on mobile GPUs, outperforming state-of-the-art approaches in accuracy, speed, and efficiency.

ABSTRACT

Recurrent neural networks (RNNs) based automatic speech recognition has nowadays become prevalent on mobile devices such as smart phones. However, previous RNN compression techniques either suffer from hardware performance overhead due to irregularity or significant accuracy loss due to the preserved regularity for hardware friendliness. In this work, we propose RTMobile that leverages both a novel block-based pruning approach and compiler optimizations to accelerate RNN inference on mobile devices. Our proposed RTMobile is the first work that can achieve real-time RNN inference on mobile platforms. Experimental results demonstrate that RTMobile can significantly outperform existing RNN hardware acceleration methods in terms of inference accuracy and time. Compared with prior work on FPGA, RTMobile using Adreno 640 embedded GPU on GRU can improve the energy-efficiency by about 40$ imes$ while maintaining the same inference time.

Motivation & Objective

• To address the challenge of achieving real-time RNN inference on mobile devices due to high computational complexity and memory consumption.
• To overcome the trade-off between model compression, inference accuracy, and hardware efficiency in existing RNN acceleration techniques.
• To develop an end-to-end framework that enables both high accuracy and high computational efficiency for RNNs on mobile platforms.
• To eliminate the limitations of existing DNN acceleration frameworks like TVM, which do not support RNNs.
• To achieve real-time performance on mobile GPUs and CPUs while maintaining high energy efficiency and model accuracy.

Proposed method

• Proposes a novel block-based structured pruning (BSP) algorithm that enables fine-grained pruning while preserving model regularity for hardware efficiency.
• Introduces compiler-assisted optimizations including matrix reordering, redundant load elimination, and a compact data format (BSPC) for pruned models.
• Uses a block-wise pruning strategy that maintains structural regularity across filters and channels, enabling efficient memory access and parallelization on mobile GPUs.
• Employs a PyTorch-Kaldi toolkit for training and fine-tuning GRU models to ensure high accuracy post-pruning.
• Optimizes code generation and data layout at compile time to maximize GPU utilization and minimize memory stalls.
• Leverages mobile GPU (Adreno 640) and CPU platforms for evaluation, comparing performance against FPGA-based ESE and other SOTA methods.

Experimental results

Research questions

• RQ1 Can a pruning strategy achieve high compression rates without degrading RNN inference accuracy on mobile devices?
• RQ2 Can compiler-level optimizations significantly improve inference speed and energy efficiency on mobile GPUs for pruned RNNs?
• RQ3 Is it possible to achieve real-time RNN inference on mobile platforms using only GPU-based acceleration, without relying on specialized hardware like FPGAs?
• RQ4 How does the combination of fine-grained structured pruning and code generation impact the trade-off between model size, accuracy, and computational efficiency?
• RQ5 What is the upper bound of compression rate achievable with minimal accuracy loss while maintaining real-time performance on mobile GPUs?

Key findings

• RTMobile achieves over 10× model compression with no accuracy degradation, outperforming ESE and C-LSTM in both compression rate and inference accuracy.
• At a 245× compression rate, RTMobile reduces model parameters from 3.25M to 0.04M while maintaining a phone error rate (PER) of 24.20%, matching the baseline C-LSTM model’s accuracy (24.15%).
• On the Adreno 640 mobile GPU, RTMobile achieves 40× higher energy efficiency than ESE, despite ESE using a high-power FPGA platform (41W).
• RTMobile maintains the same inference time (82.7 μs) as ESE on FPGA while achieving significantly better energy efficiency due to optimized memory access and parallelization.
• The inference speedup stabilizes at compression rates above 250×, indicating that computation and memory bottlenecks are effectively mitigated by the BSP and compiler optimizations.
• The BSPC compact format and compiler optimizations reduce memory access irregularity and improve GPU utilization, enabling high GOP/s even at ultra-high compression rates.