[Paper Review] Quaternion Recurrent Neural Networks

Titouan Parcollet, Mirco Ravanelli+5 more|HAL (Le Centre pour la Communication Scientifique Directe)|Jun 12, 2018

Neural Networks and ApplicationsComputer Science42 citations

TL;DR

This paper introduces quaternion-valued recurrent neural networks (QRNN) and quaternion LSTMs (QLSTM) that model external and internal feature dependencies using quaternion algebra, achieving better phoneme recognition with far fewer parameters than real-valued RNNs/LSTMs on TIMIT (and WSJ).

ABSTRACT

Recurrent neural networks (RNNs) are powerful architectures to model sequential data, due to their capability to learn short and long-term dependencies between the basic elements of a sequence. Nonetheless, popular tasks such as speech or images recognition, involve multi-dimensional input features that are characterized by strong internal dependencies between the dimensions of the input vector. We propose a novel quaternion recurrent neural network (QRNN), alongside with a quaternion long-short term memory neural network (QLSTM), that take into account both the external relations and these internal structural dependencies with the quaternion algebra. Similarly to capsules, quaternions allow the QRNN to code internal dependencies by composing and processing multidimensional features as single entities, while the recurrent operation reveals correlations between the elements composing the sequence. We show that both QRNN and QLSTM achieve better performances than RNN and LSTM in a realistic application of automatic speech recognition. Finally, we show that QRNN and QLSTM reduce by a maximum factor of 3.3x the number of free parameters needed, compared to real-valued RNNs and LSTMs to reach better results, leading to a more compact representation of the relevant information.

Motivation & Objective

• Motivate multi-dimensional feature modeling by leveraging quaternion algebra to capture internal and external dependencies in sequential data.
• Propose QRNN and QLSTM architectures with quaternion-valued inputs, weights, and activations.
• Develop quaternion-specific learning algorithms including QBPTT and a quaternion-aware parameter initialization.
• Demonstrate performance and parameter efficiency gains on speech recognition benchmarks (TIMIT, WSJ).
• Highlight potential benefits for resource-constrained devices due to reduced parameter counts.

Proposed method

• Extend RNNs to quaternion-valued computations using Hamilton product for layer transformations.
• Use split activations where each quaternion component (real, i, j, k) is activated separately.
• Apply quaternion backpropagation through time (QBPTT) to compute gradients across quaternion-valued weights.
• Introduce a quaternion-specific weight initialization procedure based on polar form and quaternion normalization.
• Derive quaternion LSTM equations with component-wise gates (QLSTM) and bidirectional training settings.

Experimental results

Research questions

• RQ1 Can quaternion-valued RNNs (QRNN) and QLSTMs capture intra-feature and inter-feature dependencies more efficiently than real-valued counterparts?
• RQ2 Do QRNN and QLSTM achieve competitive or superior phoneme recognition while using substantially fewer parameters?
• RQ3 How do QRNN/QLSTM perform on standard speech benchmarks (e.g., TIMIT, WSJ) compared to RNNs/LSTMs?
• RQ4 What initialization and training strategies are effective for stable quaternion-valued networks?

Key findings

• QRNN and QLSTM outperform RNN and LSTM baselines on TIMIT in terms of phoneme error rate (PER).
• Best TIMIT test PERs: QRNN 18.5% and QLSTM 15.1%, vs RNN 19.0% and LSTM 15.3%.
• QRNN and QLSTM achieve similar or better performance with up to 3.3x fewer trainable parameters than their real-valued counterparts.
• QRNNs and QLSTMs show substantial parameter efficiency across architectures, e.g., 4- to 16-fold reductions in some configurations.
• Across models, quaternion variants maintain competitive PER while using a fraction of the parameters (e.g., QRNN with 256 neurons: 3.8M params vs RNN 9.4M; QLSTM with 256 neurons: 14.4M vs LSTM 46.2M).
• Table 1 and Table 2 report detailed PER and parameter counts for multiple configurations on TIMIT.