[Paper Review] Compressing LSTMs into CNNs.

Krzysztof J. Geras, Abdelrahman Mohamed+7 more|arXiv (Cornell University)|Nov 19, 2015

Speech Recognition and Synthesis4 citations

TL;DR

This paper proposes compressing LSTMs into deep convolutional neural networks (CNNs) using a novel model compression technique called model blending, where a CNN student is trained under guidance from an LSTM teacher. The method achieves higher accuracy than both standalone LSTMs and CNNs, with less than 1% of the highest-probability labels needed for effective knowledge distillation.

ABSTRACT

We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs. First, we show that a deep CNN with an architecture inspired by the models recently introduced in image recognition can yield better accuracy than previous convolutional and LSTM networks on the standard 309h Switchboard automatic speech recognition task. Then we show that even more accurate CNNs can be trained under the guidance of LSTMs using a variant of model compression, which we call model blending because the teacher and student models are similar in complexity but different in inductive bias. Blending further improves the accuracy of our CNN, yielding a computationally efficient model of accuracy higher than any of the other individual models. Examining the effect of dark knowledge in this model compression task, we find that less than 1% of the highest probability labels are needed for accurate model compression.

Motivation & Objective

• To investigate whether deep CNNs can match or exceed the performance of LSTMs in sequence modeling tasks like automatic speech recognition.
• To explore model compression techniques that transfer knowledge from LSTMs to CNNs, leveraging their complementary inductive biases.
• To evaluate the effectiveness of knowledge distillation in low-label regimes, particularly using only the top probability predictions from the teacher model.

Proposed method

• Training a deep CNN with an architecture inspired by state-of-the-art image recognition models for the Switchboard 309h speech recognition task.
• Introducing model blending—a variant of knowledge distillation—where a CNN student is trained using soft labels from an LSTM teacher model.
• Applying dark knowledge distillation by using only the top 1% of highest-probability labels from the LSTM to guide the CNN's training.
• Optimizing the CNN to match the LSTM's decision function while maintaining computational efficiency.
• Using a loss function that aligns the student's output distribution with the teacher's, focusing on high-confidence predictions.

Experimental results

Research questions

• RQ1 Can deep CNNs achieve higher accuracy than LSTMs on the Switchboard 309h automatic speech recognition task?
• RQ2 Does model blending—training a CNN under guidance from an LSTM—improve performance beyond individual models?
• RQ3 How effective is knowledge distillation when only a small fraction of the teacher's top predictions are used?
• RQ4 What is the impact of inductive bias differences between LSTMs and CNNs on model compression performance?

Key findings

• The proposed CNN architecture alone outperforms previous convolutional and LSTM networks on the Switchboard 309h ASR task.
• Model blending with an LSTM teacher further improves the CNN's accuracy beyond that of the individual models.
• Using only the top 1% of highest-probability labels from the LSTM teacher enables highly accurate model compression.
• The resulting CNN model achieves state-of-the-art accuracy while remaining computationally efficient.