[Paper Review] Tensor-Train Long Short-Term Memory for Monaural Speech Enhancement

Suman Samui, Indrajit Chakrabarti+1 more|arXiv (Cornell University)|Dec 25, 2018

Speech and Audio Processing14 references4 citations

TL;DR

This paper proposes a Tensor-Train Long Short-Term Memory (TT-LSTM) model to compress deep LSTM networks for monaural speech enhancement, reducing model complexity by factorizing weight matrices using Tensor-Train decomposition. Despite being orders of magnitude less parameterized than standard RNNs, the TT-LSTM-based TensorNet achieves competitive speech quality and intelligibility performance on both matched and mismatched noise conditions.

ABSTRACT

In recent years, Long Short-Term Memory (LSTM) has become a popular choice for speech separation and speech enhancement task. The capability of LSTM network can be enhanced by widening and adding more layers. However, this would introduce millions of parameters in the network and also increase the requirement of computational resources. These limitations hinders the efficient implementation of RNN models in low-end devices such as mobile phones and embedded systems with limited memory. To overcome these issues, we proposed to use an efficient alternative approach of reducing parameters by representing the weight matrix parameters of LSTM based on Tensor-Train (TT) format. We called this Tensor-Train factorized LSTM as TT-LSTM model. Based on this TT-LSTM units, we proposed a deep TensorNet model for single-channel speech enhancement task. Experimental results in various test conditions and in terms of standard speech quality and intelligibility metrics, demonstrated that the proposed deep TT-LSTM based speech enhancement framework can achieve competitive performances with the state-of-the-art uncompressed RNN model, even though the proposed model architecture is orders of magnitude less complex.

Motivation & Objective

• Address the high parameter count and computational cost of deep LSTM models in monaural speech enhancement.
• Overcome deployment limitations on low-end devices such as mobile phones and embedded systems due to memory and compute constraints.
• Maintain high speech quality and intelligibility while drastically reducing model complexity through parameter-efficient weight matrix factorization.
• Demonstrate robustness to unseen noise types through speaker and noise generalization in evaluation.
• Develop a deep neural network framework using TT-LSTM units that preserves performance despite significant compression.

Proposed method

• Apply Tensor-Train (TT) decomposition to factorize the dense weight matrices of LSTM units into a sequence of low-rank core tensors.
• Represent each LSTM weight matrix as a chain of matrix multiplications via TT format, reducing the number of parameters while preserving representational capacity.
• Construct a deep three-layer TensorNet architecture using TT-LSTM units for end-to-end monaural speech enhancement.
• Adopt a time-frequency (T-F) masking approach, where the network predicts a mask to reconstruct clean speech from noisy input.
• Train the model using back-propagation through time with dropout (0.5) and batch normalization to improve generalization.
• Use TT-ranks (e.g., rank 4) to control compression level and balance model complexity and performance.

Experimental results

Research questions

• RQ1 Can Tensor-Train decomposition effectively compress LSTM models for monaural speech enhancement without sacrificing performance?
• RQ2 How does the TT-LSTM model perform in terms of speech quality and intelligibility compared to standard uncompressed LSTM and DNN baselines?
• RQ3 To what extent does the TT-LSTM model generalize to unseen noise types not present during training?
• RQ4 How does the model’s parameter efficiency compare to standard RNNs, and does this enable deployment on low-resource devices?
• RQ5 Does the TT-LSTM framework maintain robustness across varying SNR levels and diverse speaker populations?

Key findings

• The TT-LSTM-based TensorNet achieves PESQ and STOI scores comparable to state-of-the-art uncompressed RNN models, despite having orders of magnitude fewer parameters.
• On Test Set A (matched noise), the TT-LSTM-Rank-4 model achieves a mean PESQ of 2.45 and STOI of 0.82, outperforming the DNN baseline and matching the LSTM baseline.
• On Test Set B (mismatched noise), the TT-LSTM model maintains strong performance with a mean PESQ of 2.38 and STOI of 0.80, demonstrating robust generalization to unseen noise types.
• The TT-LSTM model reduces model complexity by multiple orders of magnitude compared to standard LSTM, enabling efficient deployment on memory-constrained devices.
• The proposed framework shows significant speaker generalization, as test speakers are entirely different from training and validation sets.
• The use of TT-ranks (e.g., 4) allows effective control of model complexity while maintaining high performance across diverse noise conditions.