[Paper Review] Efficient Speech Representation Learning with Low-Bit Quantization

Ching-Feng Yeh, Wei-Ning Hsu+2 more|arXiv (Cornell University)|Dec 14, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes efficient speech representation learning via low-bit quantization, applying robustly binarized Transformer (BiT) and squashed weight quantization to HuBERT models. It achieves 86.32% storage reduction (184.42→25.23 MB) and 88% runtime reduction (1.00→0.12) with 1-bit quantization, outperforming DistillHuBERT in efficiency and accuracy at 2-bit settings.

ABSTRACT

With the development of hardware for machine learning, newer models often come at the cost of both increased sizes and computational complexity. In effort to improve the efficiency for these models, we apply and investigate recent quantization techniques on speech representation learning models. The quantization techniques were evaluated on the SUPERB benchmark. On the ASR task, with aggressive quantization to 1 bit, we achieved 86.32% storage reduction (184.42 -> 25.23), 88% estimated runtime reduction (1.00 -> 0.12) with increased word error rate (7.06 -> 15.96). In comparison with DistillHuBERT which also aims for model compression, the 2-bit configuration yielded slightly smaller storage (35.84 vs. 46.98), better word error rate (12.68 vs. 13.37) and more efficient estimated runtime (0.15 vs. 0.73).

Motivation & Objective

• To improve model efficiency in speech representation learning by applying low-bit quantization techniques to reduce storage and computational costs.
• To investigate the trade-off between model efficiency and performance degradation under aggressive quantization, especially for edge deployment.
• To evaluate whether quantization, particularly binarization, can achieve better efficiency than existing compression methods like knowledge distillation.
• To determine the effectiveness of quantization-aware training (QAT) and knowledge distillation in mitigating performance loss during low-bit quantization.
• To compare one-step versus scheduled quantization strategies for minimizing performance degradation during extreme quantization

Proposed method

• Applied quantization-aware training (QAT) with straight-through estimator (STE) to enable gradient backpropagation through discrete quantization operations.
• Used robustly binarized Transformer (BiT) to quantize both weights and activations to 1-bit, enabling efficient inference on integer-only hardware.
• Employed squashed weight quantization to reduce gradient vanishing and improve training stability in low-precision regimes.
• Integrated knowledge distillation to transfer knowledge from a full-precision HuBERT model to the quantized model, minimizing intermediate layer and output discrepancies.
• Compared one-step quantization (direct fp16 → target bit) with scheduled quantization (multi-step reduction through intermediate precisions) to assess training stability.
• Evaluated models on the SUPERB benchmark, measuring storage, runtime, and word error rate (WER) across ASR and other speech tasks.

Experimental results

Research questions

• RQ1 Can aggressive 1-bit quantization achieve significant storage and runtime savings in speech representation models without excessive performance degradation?
• RQ2 How does low-bit quantization compare to existing model compression methods like DistillHuBERT in terms of model size, inference speed, and ASR accuracy?
• RQ3 Does knowledge distillation effectively reduce performance degradation in quantized models by aligning intermediate layer representations?
• RQ4 Is scheduled quantization (multi-step reduction) more effective than one-step quantization in preserving model accuracy during extreme low-bit compression?
• RQ5 What is the impact of combining QAT with STE and squashed weight quantization on the robustness of binarized speech models?

Key findings

• 1-bit quantization reduced model storage from 184.42 MB to 25.23 MB, achieving 86.32% storage savings on the HuBERT model.
• Estimated runtime was reduced by 88%, from 1.00 to 0.12, demonstrating significant inference speedup with 1-bit quantization.
• The 2-bit BiT-LA-w2a2 model achieved a word error rate of 12.68%, outperforming DistillHuBERT (13.37) at comparable model size.
• The 2-bit BiT-LA-w2a2 model used 35.84 MB storage, smaller than DistillHuBERT’s 46.98 MB, and achieved faster inference (0.15 vs. 0.73 estimated runtime).
• Knowledge distillation significantly improved quantized model performance, reducing WER from 18.93 (HuBERT loss) to 15.96 (knowledge distillation) at 1-bit.
• Scheduled quantization did not yield significant performance improvements over one-step quantization, suggesting that direct quantization to target bitwidth is sufficient.