[Paper Review] Low-rank passthrough neural networks

Antonio Valerio Miceli Barone|arXiv (Cornell University)|Mar 10, 2016

Neural Networks and Applications39 references6 citations

TL;DR

This paper proposes low-rank and low-rank plus diagonal matrix parametrizations for passthrough neural networks—such as LSTMs and GRUs—to decouple state size from parameter count, reducing memory and data complexity while preserving memory capacity. The method achieves a near state-of-the-art result on sequential randomly-permuted MNIST, demonstrating significant parameter efficiency without sacrificing performance.

ABSTRACT

Various common deep learning architectures, such as LSTMs, GRUs, Resnets and Highway Networks, employ state passthrough connections that support training with high feed-forward depth or recurrence over many time steps. These "Passthrough Networks" architectures also enable the decoupling of the network state size from the number of parameters of the network, a possibility has been studied by ewcite{Sak2014} with their low-rank parametrization of the LSTM. In this work we extend this line of research, proposing effective, low-rank and low-rank plus diagonal matrix parametrizations for Passthrough Networks which exploit this decoupling property, reducing the data complexity and memory requirements of the network while preserving its memory capacity. This is particularly beneficial in low-resource settings as it supports expressive models with a compact parametrization less susceptible to overfitting. We present competitive experimental results on several tasks, including language modeling and a near state of the art result on sequential randomly-permuted MNIST classification, a hard task on natural data.

Motivation & Objective

• To address the trade-off between model capacity and parameter count in deep neural networks.
• To exploit the state passthrough mechanism in architectures like LSTMs and GRUs to decouple state size from the number of trainable parameters.
• To develop efficient, low-dimensional parametrizations—specifically low-rank and low-rank plus diagonal matrix decompositions—for hidden layer transformations.
• To evaluate the proposed method on sequence modeling tasks, including permuted MNIST and character-level language modeling, with a focus on parameter efficiency and performance.
• To demonstrate that reduced parameterization does not compromise memory capacity or predictive performance when properly structured.

Proposed method

• The paper applies low-rank matrix decomposition to the recurrent weight matrices in GRU and LSTM layers, reducing the number of parameters from O(n²) to O(rn), where r is the rank.
• It introduces a low-rank plus diagonal parametrization, where the transformation matrix is decomposed as a low-rank matrix plus a diagonal matrix, enabling more expressive representations with fewer parameters.
• The method preserves the state passthrough mechanism, allowing the hidden state to flow through layers largely unchanged, thus mitigating vanishing gradient issues.
• The approach is applied to both GRU and LSTM architectures, with ablation studies on tied vs. untied projection matrices and varying rank and state size.
• Experiments use standard benchmarks including permuted MNIST and character-level language modeling, with training and evaluation protocols aligned to prior state-of-the-art work.
• Regularization via Bayesian recurrent dropout is applied to improve generalization, particularly in low-parameter regimes.

Experimental results

Research questions

• RQ1 Can low-rank parametrizations maintain high performance in deep sequential models while drastically reducing the number of parameters?
• RQ2 Does the low-rank plus diagonal decomposition offer a better inductive bias than standard low-rank or full-rank matrices for recurrent networks?
• RQ3 To what extent can the memory capacity of a network be preserved when the number of parameters is reduced via structured matrix decomposition?
• RQ4 How does the proposed method compare to existing parameter-efficient architectures like Highway Networks or ResNets in terms of accuracy and efficiency?
• RQ5 Can the method achieve competitive results on challenging tasks like randomly-permuted MNIST with a parameter count orders of magnitude smaller than standard models?

Key findings

• The low-rank plus diagonal GRU with 2.54 million parameters achieved a test perplexity of 2.76 on character-level language modeling, outperforming larger baselines with similar parameter counts.
• The tied version of the low-rank plus diagonal GRU achieved a perplexity of 2.76 with only 2.54 million parameters, demonstrating improved memory capacity efficiency.
• On the sequential randomly-permuted MNIST task, the method achieved a near state-of-the-art result, indicating strong generalization on hard, non-Markovian sequential data.
• The model with a bottleneck ratio of over 100:1 (state size to rank) still achieved strong performance, suggesting that low-rank parametrization can preserve effective representational capacity.
• The low-rank plus diagonal parametrization outperformed standard low-rank and full-rank baselines, especially in low-parameter regimes, indicating its effectiveness in parameter-efficient learning.
• Even with significantly fewer parameters (e.g., 0.46M vs. 3.11M), the low-rank GRU matched the baseline perplexity of 2.92, showing that parameter count can be reduced without performance loss.