[Paper Review] Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks

Yi Tay, Aston Zhang+6 more|arXiv (Cornell University)|Jun 11, 2019

Topic ModelingComputer Science40 references17 citations

TL;DR

This paper proposes Quaternion neural networks for lightweight and efficient natural language processing by leveraging hypercomplex number systems to reduce model parameters by up to 75% without sacrificing performance. By replacing standard real-valued operations with Hamilton products in attention mechanisms and feed-forward layers, the method enables expressive, parameter-efficient models like the Quaternion Transformer and Quaternion attention model, achieving state-of-the-art efficiency across diverse NLP tasks.

ABSTRACT

Many state-of-the-art neural models for NLP are heavily parameterized and thus memory inefficient. This paper proposes a series of lightweight and memory efficient neural architectures for a potpourri of natural language processing (NLP) tasks. To this end, our models exploit computation using Quaternion algebra and hypercomplex spaces, enabling not only expressive inter-component interactions but also significantly ($75\%$) reduced parameter size due to lesser degrees of freedom in the Hamilton product. We propose Quaternion variants of models, giving rise to new architectures such as the Quaternion attention Model and Quaternion Transformer. Extensive experiments on a battery of NLP tasks demonstrates the utility of proposed Quaternion-inspired models, enabling up to $75\%$ reduction in parameter size without significant loss in performance.

Motivation & Objective

• To address the high memory and computational cost of state-of-the-art NLP models, which often contain millions or billions of parameters.
• To explore hypercomplex representations in Quaternion space as a novel inductive bias for neural NLP architectures.
• To design parameter-efficient variants of attention and Transformer models using Quaternion algebra, enabling significant compression.
• To evaluate the proposed Quaternion models across a broad range of NLP tasks, including text classification, machine translation, and syntactic probing.
• To demonstrate that Quaternion-based models maintain or improve performance while drastically reducing parameter count.

Proposed method

• The paper introduces Quaternion representations using four-component hypercomplex numbers (r + xi + yj + zk), where i² = j² = k² = ijk = -1.
• It employs the Hamilton product (⊗) as the core operation, which encodes interdependencies between real and imaginary components, reducing degrees of freedom and enabling model compression.
• The Quaternion attention mechanism replaces standard dot products with Hamilton products to compute attention scores in hypercomplex space, enhancing expressiveness.
• The Quaternion Transformer replaces standard linear layers with Quaternion feed-forward networks, maintaining model capacity while reducing parameter count by up to 75%.
• The method supports both full and partial Quaternion adaptation, allowing flexible trade-offs between efficiency and performance.
• Models are trained using the Tensor2Tensor framework with standard hyperparameters, and the approach is compatible with real-valued components and existing architectures.

Experimental results

Research questions

• RQ1 Can Quaternion algebra be effectively applied to neural NLP models to reduce parameter count while preserving performance?
• RQ2 How does the Hamilton product in hypercomplex space improve model expressiveness compared to standard real-valued operations in attention and feed-forward layers?
• RQ3 To what extent can Quaternion-based models compress Transformer and attention mechanisms without performance degradation?
• RQ4 How do Quaternion models compare to real-valued counterparts across diverse NLP tasks, including low-resource and structured prediction settings?
• RQ5 Can the four-component structure of Quaternions naturally model multi-sense or multi-headed representations in NLP, and if so, how does it improve learning?

Key findings

• The Quaternion Transformer achieves comparable or better performance than standard Transformers on the subject-verb agreement (SVA) task, with a 75% reduction in parameters when fully compressed.
• On the SVA task, the partial Quaternion adaptation improved performance by +0.7% accuracy while reducing parameters by 25%.
• Across 13 datasets spanning 8 NLP tasks, Quaternion models maintained or exceeded the performance of real-valued baselines with significantly reduced parameter counts.
• The proposed Quaternion attention model and Quaternion Transformer demonstrated consistent efficiency gains, with up to 75% parameter reduction across all evaluated tasks.
• Quaternion components are self-contained and interoperable with real-valued components, enabling modular integration into existing models.
• The method’s effectiveness is attributed to the Hamilton product’s ability to encode latent interactions between components, increasing expressiveness while reducing parameter count.